Training large language models (LLMs) at massive scale creates distinct infrastructure challenges: jobs can span thousands of GPUs and run for long periods. The longer a training run continues, the higher the chance of unscheduled interruptions or resource fluctuations; even rare device unavailability can disproportionately slow a tightly coupled cluster. For large‑scale jobs, elastic adaptation to the number of available GPUs is an effective way to improve Goodput — the amount of useful, convergence‑driving work completed rather than raw hardware throughput.
Current elastic strategies include dropping a data replica, fast checkpoint‑restarts, or switching to hot spares. These let training jobs react to GPU availability changes while keeping system balance, but they still incur throughput losses and higher cost while operating in a reduced‑availability state.
A recent paper presents Nonuniform Tensor Parallelism (NTP), an experimental framework that extends these approaches while minimizing throughput overheads. Combined with optional dynamic power boosting to offset performance loss, NTP aims to keep throughput steady and to turn interruptions into manageable, recoverable events.
What NTP delivers
NTP’s core contribution is to prevent transient device problems from stalling large, tightly interconnected training jobs, thereby sustaining high Goodput. The approach dynamically adjusts the tensor‑parallel degree and intelligently overlaps required tensor resharding with computation phases, reducing lost time and wasted compute.
As a result, a cluster spends the maximum fraction of operating time performing useful work, preserving training efficiency and integrity as hardware conditions fluctuate.
Challenges in large‑scale training
Model training is a parallel process that often involves thousands of GPUs. A typical parallelization method is tensor parallelism (TP), where neural network layers are split across a tightly coupled GPU group. The size of that group corresponds to the scale‑up domain, which is connected by high‑speed interconnects such as NVIDIA NVLink. On NVIDIA Blackwell and NVIDIA Blackwell Ultra systems, NVLink can connect up to 72 GPUs at 1,800 GB/s, supporting all‑to‑all communication in a single hop.
A frontier LLM is often trained across multiple racks; a single rack forms a scale‑up domain and a TP group. Data parallelism (DP) replicates the model across several such domains, each processing a different batch. A change in GPU status inside a scale‑up domain can reduce a TP group’s efficiency because GPUs in the same group share tightly coupled computations: an issue with one device may slow synchronization or processing across the entire group.
As data center architectures move from smaller domains (for example, 8 GPUs) to much larger ones (72 GPUs and beyond), maximizing the productive uptime of every healthy device becomes a central factor for high Goodput.
How NTP maintains training progress
NTP is designed so that a DP replica remains productive during transient hardware interruptions. When resuming from the latest checkpoint, the model automatically adapts its configuration to the available hardware and retains partial functionality so the pipeline can keep moving without completely losing that replica’s contribution.
Key mechanisms are:
Dynamic TP‑degree adaptation
If a GPU drops out of a scale‑up domain, the system detects the affected TP group and automatically reconfigures tensor parallelism to use only the remaining functional GPUs. For example, an eight‑GPU TP group can switch to a TP degree of seven after a single dropout. The shard keeps computing, preventing total loss of the replica’s work; the remaining GPUs take on increased per‑device load, helping to sustain high Goodput and availability.
Power boosting to compensate performance
Reducing TP degree alone does not preserve global throughput: a DP replica with fewer GPUs will run more slowly and can hold back the entire DP system. To mitigate this, the paper proposes a rack design with enhanced electrical and thermal headroom that enables power boosting of affected scale‑up domains. By temporarily increasing the power to active GPUs, clock frequencies and computational throughput can be elevated.
The reduced‑TP replica can thus catch up and remain in step with fully functional replicas, avoiding global synchronization bottlenecks and helping the cluster maintain optimized Goodput despite localized hardware fluctuation.
Efficient resharding
Dynamically changing the TP degree requires redistributing model tensor shards among the remaining GPUs. NTP uses a resharding technique overlapped with other compute stages: resharding is performed during backward computation and parameter synchronization. This overlap minimizes overhead on healthy replicas, often to less than 1% extra cost. Careful scheduling keeps compute efficiency high so the adaptation itself does not become a performance bottleneck.
Hardware‑software co‑design and next steps
The work highlights the importance of co‑designing hardware and software to meet the fault‑tolerance challenges of large‑scale AI training. Integrating NTP with advanced rack designs that provide the electrical and thermal capacity for dynamic power boosting exemplifies how hardware innovations can complement sophisticated software solutions.
This symbiosis is essential to unlock greater performance, efficiency, steady Goodput and resilience in next‑generation AI systems. As an experimental feature, NTP shows how resilience can be embedded directly into parallelism strategies.
Research is already extending these concepts to Nonuniform Expert Parallelism (NEP) to improve resilience for Mixture‑of‑Experts (MoE) models where traditional tensor parallelism is less suitable.
Where to learn more
The paper points to production‑ready fault tolerance and resiliency features available in NVIDIA Resiliency Extension (NVRx) and the Nonuniform Tensor Parallelism Readme. NTP has been added to the developer branch of NVIDIA Megatron Core for developers and researchers to explore.
