The NVIDIA Rubin GPU is presented as the compute foundation of the Vera Rubin platform, engineered specifically to run "agentic" AI workloads — continuous, multi‑step reasoning and generation pipelines that plan, use tools, verify intermediate results, and operate across long contexts. NVIDIA says Rubin can deliver up to 10× more agentic throughput per unit of energy compared with NVIDIA Blackwell.
Architecture and hardware highlights
Rubin is built from two reticle‑limited compute dies joined on a single package via the NVIDIA High‑Bandwidth Interface (NV‑HBI). The chip contains 336 billion transistors, 224 streaming multiprocessors (SMs) and 896 Tensor Cores. A third‑generation Transformer Engine provides adaptive numeric precision and enables up to 50 petaflops of NVFP4 inference performance.
Compute resources are organized into Graphics Processor Clusters (GPCs) with a large centralized L2 cache. The GigaThread Engine coordinates work distribution, MIG Control supports GPU partitioning across workloads, and NV‑DEC accelerates decoding. These elements are intended to convert high compute density into sustained utilization across the diverse phases of agentic workloads.
Memory, bandwidth and I/O
Rubin supports up to 288 GB of on‑package HBM4 memory organized with 12‑Hi stacks and dedicated HBM controllers, delivering up to 22 TB/s of peak memory bandwidth — a stated 2.8× improvement over Blackwell. The enhanced Tensor Memory Accelerator (TMA) helps manage complex data layouts and descriptor handling to reduce metadata and data‑movement overhead.
Scale‑up interconnects include NVIDIA NVLink 6 with 3,600 GB/s scale‑up bandwidth to the NVLink Switch for all‑to‑all GPU communication, NVLink‑C2C at 1,800 GB/s for coherent CPU–GPU traffic, and x16 PCIe Gen 6 host connectivity up to 256 GB/s.
Reducing critical inference overheads
NVIDIA describes several features designed to shorten end‑to‑end inference paths, not just increase peak arithmetic throughput:
-
MoE weight and token movement: TMA inline descriptor updates let kernels reuse a single descriptor for tensors that share layout while overriding pointer and stride fields at runtime, reducing metadata management and movement for Mixture‑of‑Experts models.
-
Doubling K‑dimension throughput: Rubin doubles the Tensor Core throughput per clock along the K dimension, which reduces the number of K‑loop iterations in GEMMs and improves Tensor Core utilization for both throughput‑ and memory/latency‑bound kernels when execution is partitioned across many GPUs.
-
Attention on long contexts: Rubin combines activation sparsity, adaptive compression and higher softmax throughput to lower work in the QK^T → softmax → V chain. A structured 2:4 sparse compression of intermediate attention scores cuts write and storage costs while preserving the dense output expected by the rest of the model. Exponential throughput for softmax is increased (Rubin listed as 2× FP32 and 4× BF16/FP16 versus Blackwell), reducing softmax bottlenecks as Tensor Core rates rise.
-
Kernel execution efficiency: Rubin supports finer‑grained producer‑consumer coordination (tile‑level triggering) so consumer kernels can begin processing earlier as required inputs become available, tightening GPU timelines and reducing idle gaps between dependent kernels.
Scale‑up communication primitives
Rubin introduces counted writes for device‑initiated NVLink transfers. By enabling the receiving GPU to track transfer completion more directly, counted writes reduce synchronization overhead for frequent, fused GPU‑to‑GPU communications and help keep computation flowing during distributed inference.
Memory subsystem co‑design and generation scaling
Because token generation (decode) is memory‑subsystem bound, Rubin’s combination of high capacity (up to 288 GB HBM4) and high bandwidth (up to 22 TB/s) aims to host larger models and longer KV caches on‑package, reducing the need for KV‑cache offload and supporting high‑concurrency, long‑horizon inference.
TMA and memory locality strategies are described as software‑oriented mechanisms to increase achieved bandwidth for complex data layouts, an important factor for decode latency and tokens‑per‑second.
System‑level efficiency, resilience and rack design
NVIDIA extends Rubin into the Vera Rubin NVL72 rack, integrating compute, switching, 45°C liquid cooling, dynamic rack‑scale power steering and Intelligent Power Smoothing with energy storage. The aim is to treat the data center or rack as a single execution domain and recover stranded power caused by transient peaks in AI workloads.
According to NVIDIA, Intelligent Power Smoothing reduces average power consumption by approximately 10% compared with prior power‑smoothing techniques and reduces 50 ms peak power by about 20%. The DSX MaxLPS approach and DSX OS operating layer are intended to enable operators to provision up to 40% more GPUs within the same power budget at energy‑efficient operating points, while providing scheduling, lifecycle management and automation.
The Vera Rubin NVL72 rack uses third‑generation MGX architecture with cable‑free compute and switch trays, hot‑swappable NVLink switch trays, improved RAS capabilities, and flexible NVLink and NVIDIA Spectrum‑X Ethernet connectivity for multi‑rack agentic deployments.
Conclusion
NVIDIA positions the Rubin GPU and the Vera Rubin platform as a generation‑level shift for agentic AI execution: denser Tensor Cores and a faster Transformer Engine, a larger and higher‑bandwidth HBM4 subsystem, improved memory‑movement primitives (TMA), communication features (counted writes, NVLink 6), finer kernel coordination, and system‑level power smoothing. Together these elements are intended to increase useful tokens‑per‑watt and to sustain continuous, low‑latency agentic inference across larger models and rack‑scale domains.
