NVIDIA has observed that two AI clusters built from identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems can exhibit materially different training throughput—even on the same workload, model, and global batch size—with typical gaps of 8–12% and, in some cases, up to 31%. These differences most often arise from a stack of configuration choices across kernel, hypervisor, BIOS, and NCCL settings that compound and can prevent meeting the 95% threshold required for NVIDIA Exemplar Cloud validation.
This article walks through four real partner cluster debug case studies. Each diagnostic isolates a different layer: system memory management unit (SMMU) and page-table behavior on NVIDIA Grace CPU; CPU power management and NUMA placement on x86 CPUs; NCCL queue‑pair concurrency on 1.6 Tbps fabrics; and missing runtime topology or configuration inside containers. For each case we show the profiler signal (perf, NVIDIA Nsight Systems, or nccl-tests) that pointed to the root cause and the tuning change that closed most of the gap.
Prerequisites to reproduce diagnostics
- A cluster of NVIDIA HGX H100, HGX H200, HGX B200, GB200 NVL72, or GB300 NVL72 systems with NVIDIA Quantum InfiniBand or RoCE interconnect.
- A distributed training workload with stable iteration timing (examples: NVIDIA NeMo on Llama 3, NVIDIA Nemotron, or DeepSeek).
- Root access on at least one node to run perf, change BIOS/UEFI, and tune kernel parameters.
- nccl-tests built against the same NCCL version used by the training stack, NVIDIA Nsight Systems, and Linux perf with kernel symbols.
Common patterns behind training performance gaps
- Grace and virtualization readiness: missing platform capabilities, SMMU overhead, IOMMU behavior, or mismatched page-size settings.
- CPU power and process placement: cores limited below turbo frequency, wrong NUMA/PCT bindings, or helper threads pinned to training cores.
- Runtime topology: topology files or NCCL environment variables visible on the node but not inside the container/launcher.
- Fabric and collective behavior: NCCL settings that don't match the fabric, message sizes, or scale.
- Application-to-platform binding: processes bound by core ID or rank order instead of topology‑aware affinity.
These patterns are common but not exhaustive; validation with real applications remains necessary.
Case study 1 — GB200 NVL72: FP8 MoE pre-training 12% slower in VM than bare metal
Layer: Virtualization and SMMU
A partner running DeepSeek‑V3 Mixture‑of‑Experts FP8 pre‑training inside a VM on GB200 NVL72 observed 12–14% longer iteration times than the bare‑metal reference. Dense model recipes (e.g., Llama 3 70B) were within ~3% of RA, but MoE—where each iteration issues many small kernels—was the outlier.
Signal: NVIDIA Nsight Systems traces showed much higher CPU overhead in tiny kernel regions. A 30‑second perf record -a -g capture on the host, inspected with perf report, revealed ~24% of CPU cycles in arm_smmu_cmdq_issue_cmdlist.
Root cause: Under virtualization, guest SMMU invalidations trap to the host and serialize through a single command queue, producing spinlock contention. CMDQV/VCMDQ (Command Queue Virtualization) in Arm SMMUv3 lets guests issue invalidation commands directly without VM exits.
Fix: Enable CMDQV/VCMDQ in the host kernel (tegra241‑cmdqv driver) and expose it to the guest via hypervisor support (recent QEMU/libvirt provide a cmdqv IOMMU attribute). After the change, perf no longer showed arm_smmu_cmdq_issue_cmdlist dominating and dTLB miss rates returned to bare‑metal parity; MoE iteration times closed to within RA tolerance (~12% gap eliminated).
Takeaway: Grace‑based virtualized deployments must expose the host SMMU capabilities to guests for memory‑mapping‑heavy workloads. Enabling CMDQV/VCMDQ avoids unnecessary SMMU serialization and recovers MoE performance.
Case study 2 — H100 cluster: 12% loss due to CPU contention and NUMA misbinding
Layer: CPU power and process placement
An H100 SXM5 cluster running the same NCCL version and NeMo container as NVIDIA’s HGX RA was ~12% slower for Llama 3 70B pre‑training. The problem was in user space and BIOS, not the kernel.
Signals:
- turbostat -i 1 showed busy cores pegged at 3.0 GHz while the SKU turbo rating is 3.8 GHz; idle cores remained at 3.0 GHz and stuck in C1 instead of entering C6.
- numastat -p <python_pid> showed ~18% of memory accesses went to a remote NUMA node.
Root causes:
- BIOS limited C‑states to C1 (low‑latency default), keeping idle cores from dropping to C6. That consumed package power and prevented busy cores from reaching turbo.
- Hypervisor housekeeping threads were pinned to the same physical cores as training data‑loader workers, causing sporadic 50–100 ms stalls in python threads.
Fixes:
- Allow deeper C‑states (enable C6) so idle cores free package power; this recovered ~4% of performance.
- Isolate host/hypervisor services and training processes using cpuset (e.g., host on cores 0–7 and 56–63, training on the remainder), plus appropriate numactl binding and SMT/mitigation adjustments.
Result: The 12% gap reduced to ~3%; the residual difference traced to an NCCL tuning issue covered in Case 3.
Pattern: No single change fixed the whole gap—C‑state tuning contributed the largest single gain (~4%) and process isolation/NUMA binding recovered additional performance.
Case study 3 — GB300 NVL72 with ConnectX‑8 SuperNIC under‑utilizing 1.6 Tbps fabric
Focus: ConnectX‑8 SuperNIC collective tuning
A GB300 NVL72 deployment with NVIDIA ConnectX‑8 SuperNICs (1.6 Tbps per node) showed a 31% training gap on Nemotron‑4 15B pre‑training at 512 GPUs. Single‑node throughput was healthy; the gap appeared at scale and the profiler exposed AllGather and ReduceScatter time, implicating the collective path on the ConnectX‑8 fabric.
Investigation: nccl-tests sweeps (iteration count, UCX/UCC, NUMA mapping, NVLS, NCCL versions) found the effective tuning for this workload/fabric was increasing NCCL_IB_QPS_PER_CONNECTION from 1 to 4.
Signal and effect: On the NVIDIA reference cluster, the default ran ~1.09 s/iter; with QPS=4 it improved to ~0.83 s/iter. AllGather time dropped from ~375 ms to ~262 ms; ReduceScatter from ~389 ms to ~273 ms. A separate comparison run at ~0.76 s used a different NCCL version—aligning NCCL versions further narrowed the remaining gap.
Lesson: Increasing QPS is fabric‑ and workload‑dependent. On this GB300 ConnectX‑8 workload QPS=4 improved large‑message AllGather and ReduceScatter, but on other fabrics or message‑size profiles QPS may add CPU overhead without throughput gains. Test at real message sizes and sweep the setting on the target fabric.
Case study 4 — NCCL topology variable not propagated into container
In a virtualized B200 deployment training throughput was 13–53% below reference while host nccl-tests passed. Inside the enroot container, AllGather and ReduceScatter were 2–4× slower.
Root cause: NCCL_TOPO_FILE=/etc/nccl/topo.xml was set on the VM and the file existed on the host, but neither the environment variable nor the file was mounted into the enroot container. NCCL fell back to auto‑detection inside the container, causing the 13–53% deficit.
Fix: Bind‑mount the topology file into the container (e.g., --mount type=bind,source=/etc/nccl/topo.xml,target=/etc/nccl/topo.xml) and verify the variable and file from inside the job container (echo $NCCL_TOPO_FILE && cat $NCCL_TOPO_FILE).
Lesson: Run checks from inside the same container/launcher/Slurm allocation that will run the benchmark. NCCL can fail silently to auto‑detect, so verifying topology visibility inside the job environment is the fastest sanity check.
Summary of fixes and recovered performance
Across the four case studies, targeted diagnostics and remediations recovered much of the lost throughput: SMMU CMDQV/VCMDQ exposure (~12% recovery); CPU C‑state and cpuset isolation (~9% net recovery i.e., 12→3); NCCL_IB_QPS_PER_CONNECTION tuning on CX8 (~31% recovery at scale); and bind‑mounting NCCL topology into containers (closed 13–53% gaps depending on the deployment).
Preflight checks before full‑scale training debug
- GPU and hardware health: nvidia‑smi, DCGM — sustained clock, power, thermal, NVLink bandwidth.
- Grace and VM readiness: perf, dmesg, kernel config, boot params — CMDQV support, guest page size, IOMMU passthrough, large pages.
- CPU power and placement: turbostat, lscpu, numactl, nvidia‑smi topo -m — busy‑core turbo, cpuset isolation, NUMA/PCT binding near GPUs.
- Runtime topology: env, cat $NCCL_TOPO_FILE, NCCL_DEBUG=INFO — verify topology files and NCCL vars inside the job container.
- Fabric collectives: nccl‑tests and workload traces — AllGather/ReduceScatter at workload message sizes.
- Workload tuning: Nsight Systems and workload logs for pipeline parallelism, microbatch sizing, and comm overlap only after platform issues are ruled out.
Closing remarks
Performance gaps between cloud training deployments and NVIDIA reference architectures typically accumulate from multiple small configuration issues across layers. Preflight diagnostics reduce platform risk before validation; final confirmation requires running the exact validation workloads at scale and using workload traces to debug scale‑dependent gaps.
Further resources
- GTC session on demand: S81845 — Drive Optimal Tokens per Watt on any AI Infrastructure Using Benchmarking Recipes
- NVIDIA Grace CPU Performance Tuning Guide
- NCCL Environment Variables (notably NCCL_IB_QPS_PER_CONNECTION, NCCL_IB_SPLIT_DATA_ON_QPS, NCCL_IB_HCA, NCCL_DEBUG)
- NVIDIA NeMo Framework (NGC containers for Llama 3, DeepSeek, Nemotron)
- NVIDIA Performance Explorer and Performance Benchmarking Recipes
