NVIDIA argues that optimizing the full stack — from chips and cooling to rack infrastructure and software — can materially reduce energy use and token cost in AI factories. The company outlines how GPU architectures, liquid cooling, rack‑level power management and software tools such as DSX, Dynamo and TensorRT‑LLM increase tokens per watt for both inference and large‑model training.
Why inference optimization matters
Power can represent up to 40% of an AI factory's operating expenses (OpEx), so inference throughput per watt is a primary efficiency metric: improving it directly increases the number of tokens or insights an operator can sell. At hundred‑megawatt to gigawatt scales, a few percentage points of throughput gain per megawatt can produce meaningful profit increases. NVIDIA also highlights model architecture: mixture‑of‑experts (MoE) designs like DeepSeek‑R1 can be more energy‑efficient per unit of intelligence because only a subset of experts activates per token, delivering higher task performance at similar or lower per‑token compute cost compared with dense models.
System‑level energy and performance optimization
NVIDIA states its architectures and platforms increase intelligence per watt generation over generation: across six architecture generations, inference throughput per megawatt has improved by 1,000,000×. Key components and approaches include:
- NVIDIA GB200 NVL72 rack‑scale system: dense, direct‑to‑chip liquid cooling and in‑rack power smoothing that flattens peak currents, enabling operators to deploy more GPUs safely within the same power and infrastructure budget.
- NVIDIA DSX: an open AI‑factory‑scale platform that enables dynamic power allocation, real‑time telemetry and advanced rack‑level controls to recover stranded power and raise tokens per watt.
- Numeric precision choices: narrower formats such as NVFP4 are more energy‑efficient and can achieve higher throughput at comparable accuracy to FP8; higher‑precision formats tend to be slower and more energy intensive.
- Software: NVIDIA Dynamo and NVIDIA TensorRT‑LLM translate hardware gains into real inference throughput, improving scaling and lowering costs for reasoning models.
Overall system energy use depends on computation volume, hardware efficiency, GPU utilization and the operating point on the speed/energy frontier. Accordingly, system design, removing non‑GPU bottlenecks and tuning batch size for specific use cases, memory and parallelism are essential levers.
Energy efficiency in LLM training
Large model training spreads work across many GPUs using multiple parallelization strategies. Maximizing iteration speed can greatly increase energy consumption, and imbalanced GPU workloads cause idle times when some GPUs finish early and wait for others to synchronize.
Researchers in the ML.ENERGY Initiative at the University of Michigan have shown that tuning the processing speed of individual GPUs reduces energy bloat in large model training. GPUs on the critical path (with more work) run at full speed, while less busy GPUs are intentionally slowed. This approach:
- minimizes idle time from early‑finishing GPUs
- lowers energy consumption of slowed GPUs
- keeps end‑to‑end training time effectively unchanged
NVIDIA collaborates with the ML.ENERGY team to improve Megatron‑LM (NVIDIA's open‑source reference for large‑scale language model training) energy efficiency through kernel‑ and phase‑level profiling and analyzing how parallelism, pipeline imbalance and communication overlap affect performance‑per‑watt. These measurements guide energy‑aware scheduling and GPU frequency/power‑cap tuning aligned with the true critical path of training iterations. The aim is to finish training faster within the same power envelope or to reach the same throughput with less energy, freeing power for extra training runs or inference. NVIDIA cites roughly up to 25% energy savings at similar iteration times from energy‑aware scheduling and runtime optimization.
How NVIDIA DSX optimizes AI factory performance
The ML.ENERGY Initiative has produced benchmarks and a reasoning framework to explain observed energy behaviors; these can be integrated into telemetry‑driven, energy‑aware operations. NVIDIA DSX provides such operational capabilities, giving a coordinated view across compute, racks, cooling, facility power and workload scheduling. DSX connects design‑time simulation with runtime telemetry so operators can find where power is used, where it is stranded, and how much additional useful compute fits within a fixed site envelope.
DSX defines how AI factories are designed, built and optimized across the full stack. It bundles open software libraries, workflow guides and reference designs with NVIDIA compute platforms and co‑designed OEM infrastructure to enable a broad ecosystem of solutions.
DSX delivers power management at multiple layers:
- DSX MaxLPS: in‑factory technologies to maximize throughput, including 45°C liquid cooling to improve PUE, dynamic power allocation that monitors and reallocates GPU and rack‑level consumption within defined budgets in real time, and advanced techniques embedded in NVIDIA GPUs (power steering, optimized workload profiles, and orchestration with NVIDIA Dynamo).
- DSX Flex: a grid‑aware orchestration layer that connects the AI factory to grid signals and external energy sources.
With end‑to‑end power, cooling and grid integration optimized, operators can focus on workload‑level efficiency: using benchmarks to choose models, batch sizes and precision, aligning workload placement and power allocation with the most efficient compute and cooling zones, and prioritizing workloads that justify higher power because they generate more revenue per token.
Practical impacts and outlook
NVIDIA emphasizes that power constraints make performance per watt the driver of token cost and profitability. Full‑stack improvements increase tokens per watt, accelerate deployment and strengthen reliability. The company recommends treating AI tokenomics metrics as first‑class design goals and combining digital‑twin‑driven infrastructure optimization with benchmark‑driven workload tuning so constrained power becomes a competitive advantage. NVIDIA reports that a performance‑per‑megawatt optimized AI factory can deliver up to 2.6× more tokens per second per megawatt than an unoptimized factory at target interactivity levels.
Event and acknowledgments
NVIDIA will present more on power‑constrained AI factory design, simulation and operations — and DSX — at the ISC 2026 conference. The company thanks Mosharaf Chowdhury, Jae‑Won Chung and Ruofan Wu from the ML.ENERGY Initiative at the University of Michigan for their contributions.
