SemiAnalysis’s AgentX benchmark, which replays production-style coding agent sessions, measures how efficiently accelerators serve long-context, interactive, tool-invoking workloads. NVIDIA’s preliminary NVL72 measurements show Vera Rubin delivering up to 30× higher token throughput per megawatt than GB300 NVL72 at a 160 tokens/sec per-user interactivity target on the DeepSeek V4‑Pro workload, while GB300 NVL72 retains a multi‑order-of-magnitude throughput-per-megawatt advantage over H200 NVL8 across several agentic scenarios.
What is AgentX and why it matters
AgentX is the agentic-coding benchmark within the open-source InferenceX suite from SemiAnalysis. Unlike fixed-length prompt–response tests, AgentX replays prerecorded Claude Code sessions that include multi-step reasoning, tool calls, and accumulating context. The benchmark preserves each session’s input/output lengths, timing, reasoning delays, and tool-call latency so that KV-cache pressure and realistic timing behavior are reproduced.
Because every platform receives the same recorded traffic and timing, differences in results reflect the serving stack and system design rather than benchmark-specific tuning. AgentX therefore aims to reveal how well a platform can serve replayed agent traffic responsively, reuse previously processed context, and maximize agentic throughput per provisioned megawatt.
Measurement approach and key metrics
Agentic sessions are variable and stateful: request lengths change turn-to-turn, context accumulates, tool execution interrupts model calls, and previously processed tokens can be reused. To capture these properties, AgentX replays sessions turn by turn via the AIPerf client and varies concurrency to map the trade-off between throughput and interactivity.
The primary metric for AI-factories is tokens per megawatt. AgentX reports that metric against four user-experience targets:
- E2E Normalized Interactivity: output tokens divided by the time from request submission to final-token arrival; shows user-visible output per megawatt including TTFT.
- Standard Interactivity: output tokens divided by the time from first to last token; shows streamed output rate once generation begins (excludes TTFT).
- E2E Latency: total elapsed time from submission to the final token; ensures throughput is paired with acceptable request completion time.
- TTFT (Time To First Token): time from submission to first output token; important for agents starting long-context turns frequently.
By scanning concurrency levels, AgentX reports sustained throughput per provisioned megawatt together with each stack’s ability to reuse context.
NVIDIA Vera Rubin NVL72: preliminary AgentX results
NVIDIA measured Vera Rubin NVL72 on the SemiAnalysis AgentX workload (results pending SemiAnalysis review). On the AgentX DeepSeek V4‑Pro workload at 160 tokens/sec per user, Vera Rubin NVL72 reportedly delivers up to 30× higher AI‑factory throughput per megawatt than GB300 NVL72. This suggests a substantial increase in agentic inference capacity while holding the same interactive serving target.
Further technical background on Rubin GPU design is available in NVIDIA’s Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI.
NVIDIA GB300 NVL72: AgentX results and cost implications
On AgentX, GB300 NVL72 also shows significant gains relative to previous generation hardware:
- For DeepSeek V4 Pro 1.6T, GB300 NVL72 delivers up to 15× higher token throughput per megawatt than H200 NVL8 on the AgentX workload.
- That performance advantage translates to unit economics: GB300 NVL72 can yield up to 10× lower cost per million tokens compared to H200 NVL8, enabling operators to support materially more interactive capacity or lower operating expenses under a fixed power/infrastructure budget.
The advantage grows with model scale: on Kimi K3 2.8T, GB300 NVL72 achieves roughly 80× the throughput per megawatt of H200 NVL8 at comparable interactivity, and extends the interactivity frontier to about 215 tokens/sec per user—well beyond the operating range of H200 NVL8.
System-level factors behind the results
GB300 NVL72’s gains stem from system-level work across serving runtimes, model kernels, and scale-up fabric, enabling large Mixture-of-Experts (MoE) models to sustain responsive throughput as sessions accumulate context and concurrency increases.
Key elements include:
- MoE serving runtimes: frameworks such as SGLang, TensorRT‑LLM, and vLLM distribute expert execution across the NVL72 domain. Techniques like Wide Expert Parallelism and DeepEP balance expert work across GPUs and increase effective batch size for concurrent requests.
- MoE kernels and communication overlap: DeepGEMM-based kernels, mixed-precision formats (MXFP4, MXFP8), and fused MoE execution paths reduce data movement between expert stages. Overlapping expert-parallel communication with Tensor Core computation improves token throughput for reasoning and coding loads.
- NVIDIA Dynamo: separates prefill and decode into independently scaled worker pools, supports session-aware serving via session IDs, and uses a KV-cache-aware router to reduce unnecessary prefill work—helping maintain responsive multi-turn serving as sessions reuse context.
- NVLink scale-up fabric: NVLink connects the 72 GPUs in GB300 NVL72 in a high-bandwidth rack-scale domain, enabling coordinated compute, memory, expert-parallel communication, and KV-cache movement needed for large models.
Outlook for agentic AI deployments
Vera Rubin NVL72 demonstrates the potential of hardware and system co-design targeted at long-context, interactive, distributed agentic execution. The broader Rubin approach distributes phases of the workflow to the most suitable processors—Rubin GPUs for large context and decoding, Vera CPUs for tool execution and KV-cache offload, and Groq 3 LPX for ultrafast interactivity—while fabrics and software (NVLink 6, ConnectX‑9, BlueField‑4, Spectrum‑X, Dynamo, Attention‑FFN Disaggregation, NVFP4, TensorRT‑LLM WideEP, speculative decoding) coordinate execution to reduce recomputation and waiting.
The practical goal is to sustain interactive performance as agent sessions grow and to convert a larger share of a fixed power budget into useful agentic output.
Acknowledgments
NVIDIA cites engineering contributions from Xin Li, Ankur Singh, Anthony Casagrande, Jonas Li, Po‑Han Huang, Xiaoming Chen, and other engineers for the work summarized in these results.
This article is based on the AgentX benchmark description from SemiAnalysis and preliminary NVL72 measurements provided by NVIDIA; the NVIDIA measurements are subject to SemiAnalysis review.
