NVIDIA reports that its GB300 NVL72 rack‑scale system set a world record when pre‑training the DeepSeek‑V3 671B model: using 256 GPUs the system delivered 1,648 TFLOPs per GPU. NVIDIA attributes the result to coordinated design across silicon, interconnect, networking and software and to continuous optimization of the platform.
Why this matters for MoE models
The company notes that the frontier of pre‑training has shifted toward mixture‑of‑experts (MoE) architectures because they substantially reduce compute per token. DeepSeek‑V3 contains 671 billion parameters in total, but only about 37 billion parameters are activated per token, so the per‑token cost is comparable to a much smaller dense model.
That efficiency comes with a communication trade‑off: experts are located on different GPUs, so every MoE layer must route tokens to experts and gather results using an all‑to‑all communication pattern in both forward and backward passes. Because this communication occurs at every layer and every training step, small delays accumulate; once the all‑to‑all cannot be hidden behind compute, adding GPUs stops increasing throughput.
Two‑tiered communication challenge and GB300 NVL72 design
Efficient pre‑training requires a tightly coupled scale‑up domain where every GPU can communicate over a non‑blocking fabric with uniform high bandwidth and low latency. Large models require more GPUs than a single domain can hold, so multiple domains must be linked; scale‑out traffic is lighter but must still complete predictably within the compute window. Success is measured in delivered FLOPs rather than peak FLOPs.
GB300 NVL72 is built around that two‑tiered communication requirement: rather than only a faster chip, the rack‑scale system uses extreme co‑design of silicon, interconnect, networking, infrastructure processing and software.
- NVLink (fifth generation) provides the scale‑up fabric, enabling 72 NVIDIA Blackwell Ultra GPUs to appear as a single unit. NVLink supplies 1.8 TB/s per GPU and 130 TB/s of non‑blocking all‑to‑all bandwidth across the rack so every GPU reaches every other in one hop.
- NVLink is memory‑semantic: a GPU reads and writes a peer’s HBM directly as native load/store operations over a lossless, flow‑controlled fabric. Transfers are hardware memory operations rather than software sends, minimizing added latency and enabling reductions to run inside the switch as data flows.
- For scale‑out, the platform uses NVIDIA ConnectX‑8 SuperNICs at 800 Gbps per GPU with NVIDIA Quantum‑X800 InfiniBand or NVIDIA Spectrum‑X Ethernet to keep gradient traffic hidden behind compute.
At the infrastructure level, NVIDIA BlueField DPUs provide an isolated domain for virtual networking, storage access, security, telemetry and lifecycle management, reducing host CPU overhead for large training jobs.
Software optimizations and measured results
Training software was tuned for these GPUs. Using NVIDIA Megatron Core, DeepSeek‑V3 671B achieved 1,648 TFLOPs/GPU on the GB300 NVL72 with 256 GPUs, compared to 606 TFLOPs/GPU reported earlier on GB200 NVL72 — roughly a 3× increase in delivered throughput per GPU in a single generation.
NVIDIA also reports that software improvements alone delivered a 1.5× performance increase on the same GB300 NVL72 rack over six months, indicating that training throughput continues to improve after silicon ships.
Open frameworks show similar gains:
- TorchTitan (PyTorch native training stack) received continuous NVIDIA contributions; on DeepSeek‑V3 671B these optimizations cumulatively delivered approximately 6× higher delivered performance on the same GB300 NVL72 infrastructure.
- JAX optimizations from NVIDIA yielded nearly 10× improvement over six months at 256‑GPU scale on DeepSeek‑V3 671B. The latest JAX software version reached 1,025 TFLOPs/GPU on GB300 NVL72.
In scaling tests from 256 to 1,024 GPUs, Megatron Core retained 98.5% of its per‑GPU performance while TorchTitan and JAX each retained about 97%. This indicates that most added infrastructure translates into additional system‑level tokens/second, with the 800 Gb/s per‑GPU scale‑out fabric keeping gradient traffic hidden behind compute.
Record and outlook
NVIDIA states that the 1,648 TFLOPs/GPU result on GB300 NVL72 for DeepSeek‑V3 671B allows the same pre‑training job to reach comparable performance with a smaller hardware footprint than the previous generation. The company emphasizes these results are not a limit but a foundation for further hardware and software improvements.
Acknowledgements
NVIDIA acknowledged contributions from many engineers across the company, listing individuals including Aidyn Aitzhan, Michael Andersch, Jan Bernloehr and others.
(Article content is based on the technical and numerical details provided in the source announcement; no additional dates or external verification data were supplied.)
