In agentic AI systems, High Bandwidth Memory (HBM) is the fastest, costliest and most contested memory tier. Any byte used for redundant weight copies, an idle session’s KV cache or a recomputed prefix does not produce tokens. NVIDIA frames the central question as how much useful work each memory byte delivers rather than how much memory a GPU has.
Generational data from GB300 to NVIDIA Vera Rubin shows that HBM capacity stayed roughly at 288 GB while bandwidth climbed to 19.2 TB/s and agentic throughput per megawatt increased dramatically. On the SemiAnalysis AgentX benchmark (which replays recorded coding‑agent sessions with real tool‑call gaps and growing context), Vera Rubin NVL72 delivered up to 30× the throughput per megawatt of GB300 NVL72 at comparable interactivity (for example, 160 tokens/s per user). Capacity alone cannot explain a 30× gain from the same 288 GB per GPU; the difference is how much useful work each byte performs.
NVIDIA treats this as a co‑design challenge spanning GPU architecture, the scale‑up interconnect, serving software, the CPU and storage. The company highlights five levers that together increase the value of each memory byte, from bit‑level formats to pod‑wide shared storage.
1. NVFP4: packing more intelligence into fewer bits
The most direct way to raise a byte’s value is to make each stored value smaller without losing the information it encodes. NVFP4 uses a 4‑bit E2M1 element combined with two levels of scaling: an FP8 (E4M3) scale factor shared across every 16‑value micro‑block and a per‑tensor FP32 scale above that. This yields a compact format that still preserves accuracy where naive 4‑bit formats would fail.
Applied to model weights, NVFP4 reduces footprint roughly 3.5× compared with FP16 and 1.8× compared with FP8, freeing HBM for the KV cache, larger batches and higher concurrency. NVFP4 has become the default precision for several open agentic benchmarks with strict accuracy needs (AA‑AgentPerf, SemiAnalysis AgentX, MLPerf Endpoints). Across those benchmarks and leading MoE architectures, NVFP4 can provide up to 4× the throughput of an FP8 baseline at the same interactivity.
The same principle applies to the KV cache: NVFP4 reduces the KV cache footprint by up to 50% versus FP8, so roughly twice as much context can remain resident on device. Fewer evictions lead to higher cache‑hit rates during prefill and less bandwidth consumed during decode.
2. Wide Expert Parallelism and NVLink: making 72 GPUs behave like one memory pool
MoE models activate only subsets of parameters per token, yet expert weights must be stored somewhere in HBM. Packing experts onto few GPUs leaves each GPU’s memory dominated by weights and forces expensive weight‑loading during decode. Wide Expert Parallelism (Wide‑EP) in NVIDIA TensorRT‑LLM spreads experts across eight or more GPUs, changing that balance.
For example, DeepSeek‑R1 at EP64 places four experts per layer on each GPU, distributing 232 expert blocks across 58 MoE layers instead of concentrating them. The consequences are:
- Capacity: more HBM per GPU is available for KV cache, larger batches and concurrency.
- Arithmetic intensity: GroupGEMM kernels reuse loaded weights across more tokens, increasing FLOPs per byte.
- Balance: the Expert Parallel Load Balancer (EPLB) redistributes hot and cold experts across forward passes so no GPU is overcommitted while others sit idle.
This approach relies on a high‑bandwidth scale‑up domain so that remote bytes are nearly as useful as local ones. MoE layers trigger all‑to‑all dispatch and combine traffic that must be absorbed by fabric; on GB200 NVL72, 130 TB/s aggregate NVLink bandwidth supports that traffic. Large EP (EP32) can deliver up to 1.8× more output tokens per second per GPU than small EP (EP8) at a given interactivity.
Vera Rubin NVL72 continues this design: each Rubin GPU offers up to 288 GB of HBM4 and 3.0 TB/s of NVLink 6, yielding roughly 20.7 TB of HBM4 across a rack. Rubin also adds inline memory tensor accelerator (TMA) descriptor updates so one shared descriptor can serve every expert with a common layout, reducing metadata overhead that grows with expert count.
Wide‑EP therefore decides where weight bytes live; the next lever decides whether KV cache bytes are recomputed or reused.
3. Dynamo: making every cached byte count across the fleet
The KV cache stores intermediate attention results so the model can reuse them instead of recomputing. That state is ephemeral and can be evicted and recomputed, but it is only valuable if reused. Agentic workloads offer many reuse opportunities (system prompts, tool schemas, shared documents), yet a request landing on a GPU without the needed prefix—or on a GPU that recently evicted it—forces recomputation.
NVIDIA Dynamo is the distributed serving layer that treats KV cache as a fleet‑wide resource rather than a per‑GPU side effect. Dynamo combines three co‑designed capabilities:
- KV‑aware routing: the Dynamo router tracks where KV blocks reside and considers both cached‑prefix overlap and worker load when placing requests, turning potential recomputations into cache hits.
- Disaggregated serving: prefill and decode run on separate GPU pools sized for their own compute and memory profiles; decode GPUs can dedicate HBM to KV cache and batch size rather than reserving headroom for bursty prefill.
- KV Cache Runner (KVCR): tracks where each block lives across GPU, pinned CPU and storage tiers (including local SSDs and mounted network filesystems). A prefix preserved outside HBM can be restored when later needed, letting useful context survive GPU eviction. KVCR uses the NVIDIA Inference Xfer Library (NIXL) for storage transfer paths so cached blocks can move while the GPU processes other work. KVCR also enables distributed reuse with router hints helping a worker retrieve cached context from a peer; KVCR‑Guard preserves committed host‑memory cache across an engine failure while the guard service and host remain available.
Dynamo’s KV‑cache offload depends on fast retrieval: preserving a prefix outside HBM is worthwhile only if loading it is faster than repeating prefill. NIXL connects this offload path to external storage systems. In validation tests, storage vendors VAST Data and WEKA demonstrated 35 GB/s to one H100 GPU using GPUDirect Storage and up to 270 GB/s aggregate read throughput across eight H100 GPUs through a custom NIXL plugin, respectively. Those tests showed external storage could supply KV cache without becoming the transfer bottleneck, enabling cache capacity beyond HBM. A VAST test also showed response‑time benefits: using vLLM and LMCache on a DGX H100 system, loading stored KV cache rather than recomputing reduced time‑to‑first‑token for Qwen3‑32B with a 130K‑token prompt.
Dynamo coordinates where cached bytes live and when they return. The next two levers expand the capacity available for that context, beginning with CPU memory.
4. NVIDIA Vera CPU: preventing idle context from wasting HBM
Agentic inference alternates between GPU model steps and CPU‑side work (tool calls, code execution, retrieval and evaluation). During those gaps, session KV cache kept in HBM occupies capacity without generating tokens, and other requests compete for the same bytes. A slow CPU therefore costs HBM twice: as idle residency and as recomputation later.
The NVIDIA Vera CPU addresses both sides:
- Shorter gaps: 88 Olympus cores deliver 1.8× faster single‑threaded performance, 40% lower peak loaded latency than x86 CPUs and up to 14 GB/s memory bandwidth per core. Faster, more predictable tool calls shrink the window in which KV cache is exposed to eviction.
- A fast place to park context: each Vera CPU provides up to 1.5 TB of LPDDR5X—more than 5× a Rubin GPU’s HBM4—and NVLink‑C2C connects it coherently to Rubin at 1.8 TB/s, roughly 7× the 256 GB/s of an x16 PCIe Gen 6 link.
Together, these capabilities change the economics of an idle session. Instead of holding KV cache in HBM or discarding it, Dynamo’s KV Block Manager can stage it in Vera memory for the duration of a tool call and bring it back over NVLink‑C2C when the agent resumes. HBM remains dedicated to actively generating sessions and paused sessions avoid paying the prefill cost twice.
Vera memory serves as a warm tier sized for context returning within a turn. For context that must survive across turns, agents and nodes, the AI factory needs a still larger tier.
5. CMX: extending context memory across the pod
Keeping every reusable KV‑cache byte in GPU or host memory caps capacity for active inference; discarding it forces recomputation. Multi‑turn sessions, long‑horizon tasks and multi‑agent workflows generate KV cache that will be needed again, often minutes later and on different nodes. Local SSDs are bound to a single node and enterprise shared storage brings replication and metadata overhead that ephemeral KV cache does not require.
NVIDIA CMX introduces a pod‑level G3.5 tier: Ethernet‑attached flash shared at pod level and optimized specifically for KV cache. Built on the NVIDIA STX reference design, CMX is co‑designed across four layers:
- Storage processing: BlueField‑4 STX Storage Processor manages NVMe SSDs, runs storage services and offloads data integrity and encryption for KV cache.
- Networking: NVIDIA Spectrum‑X Ethernet provides low‑latency, high‑bandwidth RDMA access between compute nodes and CMX storage nodes.
- KV‑aware services: NVIDIA DOCA Memos provides KV‑aware services and I/O control on BlueField‑4, treating context as a first‑class resource for placement and sharing.
- Serving integration: Dynamo and NIXL make CMX appear as a seamless context tier across the pod, prestaging context and routing requests to where KV cache already resides.
Pod‑level pooling means a prefix computed once on one node can be reused across compatible model instances in the pod, so agents that share documents, tools or working memory can avoid duplicate prefill. Pooling reduces KV‑cache duplication and stranded capacity across nodes, extending effective GPU memory for inference. By prestaging reusable context into GPU or host memory, CMX reduces inference stalls; its KV‑cache‑optimized storage tier also lowers the power overhead of serving that context. NVIDIA reports that CMX can deliver up to 5× higher tokens per second and up to 5× better power efficiency compared with traditional storage for long‑context inference.
How the five levers stack up
Each lever eliminates a different kind of waste: NVFP4 removes unnecessary bits, Wide‑EP removes redundant weight copies, Dynamo avoids recomputation, Vera prevents idle residency during tool calls, and CMX enables KV‑cache sharing and reuse across nodes. Because these levers are designed as a cohesive memory system, the benefits stack: a 4‑bit KV block saves space in HBM, in Vera memory and in CMX.
The result is an AI factory that measures memory by the useful tokens each byte produces. Vera Rubin NVL72 exemplifies this approach, showing up to 30× throughput per megawatt versus GB300 NVL72 from the same 288 GB per GPU. As agentic workloads grow, every byte counts.
Contributors
The work reflected engineering contributions from Wei‑ming Chen, Shiyang Chen, Chao Ni, Mengdi Wang and Jeff Pool.
