Table of Contents
The AgentX result is impressive, but it rides on one specific operating point
In sum – what we know:
- A narrow 30× – The efficiency figure is measured at 160 tokens per second on DeepSeek V4 Pro, a model already phased out in favor of DeepSeek V4.1 Flash.
- A systems benchmark – AgentX replays real recorded coding-agent traffic and measures serving throughput per megawatt, not raw GPU FLOPS or output quality.
- A vendor claim – The 30× is Nvidia’s figure on an open framework, with independent confirmation from SemiAnalysis still pending.
According to Nvidia, the Vera Rubin NVL72 rack system delivers up to a 30× higher agentic-AI throughput per megawatt than the preceding GB300 NVL72 system in SemiAnalysis’ AgentX benchmark. The company also says the comparison corresponds to up to 45× lower cost per million tokens, an updated figure from the 35× it cited in earlier materials.
According to Nvidia, the platform entered production in July and is already being deployed by cloud partners including CoreWeave, Google Cloud, Microsoft Azure, Oracle Cloud Infrastructure, and Nebius. That production timeline matters because the benchmark result is framed around a real and growing constraint in AI infrastructure. Data-center electrical capacity, not just GPU supply, is increasingly what limits how much AI-agent work an operator can actually serve. Higher throughput per megawatt theoretically lets an operator handle more work without securing additional grid capacity. For scale, SemiAnalysis cites a roughly 2,300-watt thermal design power per compute tray in these racks.
What 30x actually represents
The 30× figure is not a general-purpose performance multiplier. Nvidia’s technical disclosure ties the result to a specific configuration. It was measured at 160 output tokens per second per user on DeepSeek V4 Pro, a 1.6-trillion-parameter mixture-of-experts model, running the AgentX agentic-coding workload. One caveat: SemiAnalysis notes that at the time of writing, DeepSeek V4 Pro had been replaced in favor of DeepSeek V4.1 Flash, so the 30× rides on a model already being phased out. The metric itself is AI-factory throughput per megawatt while maintaining a specified responsiveness target — not raw GPU FLOPS, single-user token speed, or end-to-end task-completion quality.
Nvidia describes the result as a point on a performance/interactivity curve. The company says Vera Rubin improves the full curve, but the headline 30× is the best point cited at the high-interactivity end. That distinction is worth holding onto, because the number shifts depending on where you look on that curve and what latency target you’re measuring against.
The AgentX benchmark
AgentX is a new agentic-coding scenario within SemiAnalysis’ open-source InferenceX suite. It’s designed to measure something that traditional inference benchmarks largely ignore. Modern AI agents don’t fire off a single prompt and wait for a response. They make many model calls, invoke external tools, launch sub-agents, and carry growing context across long tasks. AgentX tries to capture that behavior.
Rather than submitting identical fixed-length prompts, AgentX replays real recorded coding-agent traffic turn by turn. The v1.0 corpus includes 393 Claude Code sessions, collected on June 21, 2026. Each selected session has at least 20 requests, and the full variant retains contexts up to one million tokens. SemiAnalysis says it spent more than $3 million assembling the trace dataset, and the benchmark code and trace-replay harness are publicly available.
One thing AgentX does not do is evaluate model-output quality. It measures the serving system’s ability to deliver tokens under a realistic replay of agent traffic, full stop. The benchmark also runs in a closed loop, meaning faster systems can progress farther through sampled agent sessions in the same hour. SemiAnalysis advises reporting throughput together with time-to-first-token and interactivity rather than treating a single latency number as definitive. “Concurrency” in AgentX refers to live agent clients, not a fixed HTTP batch. A single agent can fan out into several sub-agent requests, so the actual number of in-flight model requests can exceed the configured concurrency.
There are limitations baked into the methodology. The synthetic payload approach preserves lengths and cache-prefix structure but not the semantic content of original sessions. SemiAnalysis uses model-specific forced-acceptance settings for speculative decoding because synthetic tokens would otherwise have different draft-token acceptance behavior from natural outputs. AgentX also can’t reproduce hidden provider-side transformations like proprietary chat templates, tokenizers, encrypted reasoning, server-side tools, or multimodal-token expansion.
Why agentic workloads need different benchmarks
Traditional inference tests typically use fixed input and output sizes — something like an 8,000-token prompt followed by 1,000 output tokens. Those are useful for establishing a controlled hardware baseline, but they don’t simulate the long, stateful, bursty workloads that coding and research agents actually generate. In agent sessions, prior conversation and tool outputs become part of the next request, creating context windows that grow continuously over the life of a task.
That growth changes what matters most in the serving stack. Reuse of the model’s KV cache — essentially stored attention state from earlier tokens, which allows the system to avoid reprocessing the entire conversation history on every new request — becomes a major determinant of performance. The serving system has to keep or retrieve the right cache, route requests to the GPU where it already resides, and avoid recomputing context unnecessarily. Fixed-length benchmarks don’t stress any of that.
SemiAnalysis says its older fixed-length InferenceX scenario has been put into “maintenance mode” because it considers multi-turn, long-context agent workloads a more representative production test. Nvidia, for its part, cites OpenRouter data indicating an agentic request can consume roughly 15× the tokens of an ordinary chat interaction. That statistic is Nvidia’s framing of the workload shift and shouldn’t be confused with a measured outcome of AgentX itself, but it does underscore why the company is eager to benchmark against agent-shaped traffic rather than simple prompt-response pairs.
In its own August AgentX launch coverage, SemiAnalysis tested systems including the GB300 NVL72, GB200 NVL72, B300, B200, H200, and AMD’s MI355X. SemiAnalysis identified major performance differences across hardware platforms and serving stacks, including cases where AMD’s vendor-specific ATOM runtime was more competitive than upstream open-source software. That reinforces the point that AgentX results measure hardware-and-software configurations together, not just silicon.
The Vera Rubin NVL72 hardware
Vera Rubin NVL72 is a rack-scale system, not a single GPU or server. Nvidia lists 72 Rubin GPUs and 36 Vera CPUs per rack, linked through sixth-generation NVLink switching. The company specifies 20.7TB of HBM4 memory and up to 1,580 TB/s of GPU memory bandwidth per rack, along with a published spec of 3,600 PFLOPS of sparse NVFP4 inference performance — though theoretical FLOPS aren’t directly comparable with AgentX’s throughput-per-megawatt result. Nvidia describes it as part of a broader “seven-chip” platform that includes Rubin GPUs, Vera CPUs, ConnectX-9 SuperNICs, BlueField-4 DPUs, NVLink 6 switching, and Spectrum-6 SPX. The company also positions Groq 3 LPX systems as complementary infrastructure for very-low-latency inference.
Nvidia attributes the claimed efficiency gain to full-rack hardware, networking, and software co-design rather than solely to the new GPU. Several specific architectural choices drive the result, according to the company. Disaggregated prefill and decode allow context processing and token generation to be assigned to separate worker pools and scaled independently — prefill is generally compute-heavy, while decode is more memory-bandwidth-sensitive. KV-aware routing directs requests to GPUs that already hold relevant prior context, and less-active cached context can be offloaded to host memory or storage rather than recomputed.
DeepSeek V4 Pro’s mixture-of-experts architecture also plays a role here. Expert subnetworks, the specialized portions of the model that activate selectively for different inputs, can be distributed across the full 72-GPU domain. NVIDIA says NVLink 6 provides the low-latency, high-bandwidth GPU-to-GPU communication needed for expert routing and cache movement at that scale.
On the precision side, NVIDIA says 4-bit floating-point quantization (NVFP4) and the Rubin GPU’s fifth-generation Tensor Cores reduce model-memory requirements and raise inference throughput. And the software stack matters too — Nvidia cites MegaMoE CUDA kernels, TensorRT-LLM, SGLang, vLLM, and its Dynamo serving software. Because AgentX is explicitly a systems benchmark, where routing, caching, and runtime decisions can materially move the result, the software layer is arguably as important as the silicon underneath it. How much depends on the baseline: SemiAnalysis’ own data shows the gap collapses when the GB300 side runs SGLang instead of TensorRT-LLM — roughly 62.9× at matched interactivity, but only about 5.56× at 170 tokens per second and 2.09× at 100.
Separately from the AgentX result, Nvidia announced that its DSX MaxLPS power-management software can dynamically move available power across GPUs and racks. Cloud provider Lambda reported that on Nvidia Blackwell servers, it ran 19 nodes within a power budget typically sized for 16, raising aggregate token throughput from about 4 million to 5 million tokens per second — a 24% increase — and performance per watt by 23%. NVIDIA says DSX MaxLPS could enable up to 40% more GPU capacity in the same site power budget for suitable Vera Rubin deployments. That’s a separate factory-level power-provisioning claim, not part of the AgentX 30× chip and rack result, but it reinforces NVIDIA’s broader push to reframe AI-infrastructure competition around useful work per megawatt rather than peak FLOPS.
GB300 as the baseline and how it compares elsewhere
GB300 NVL72 is Nvidia’s Blackwell Ultra rack-scale system, and it’s the direct baseline for the Vera Rubin claim — not an older Hopper-era product. That’s worth emphasizing because the comparison isn’t against aging hardware. On the same DeepSeek V4 Pro AgentX workload, Nvidia says GB300 NVL72 itself offers up to 15× higher throughput per megawatt than an H200 NVL8 system. On the much larger Kimi K3, a 2.8-trillion-parameter model, Nvidia says that lead reaches roughly 80×. Both figures are separate from the Vera Rubin headline.
MLPerf Inference v6.1, a separate peer-reviewed industry benchmark suite, reported Vera Rubin NVL72 delivering up to 3.7× more throughput than GB300 NVL72 (on Qwen3-VL) in NVIDIA’s first preview submission. That result is notdirectly comparable to the 30× AgentX figure because MLPerf and AgentX measure different workloads and metrics. The gap between 3.7× and 30× is striking, but it’s explained by the specificity of the AgentX result. The much larger ratio reflects a particular long-context, multi-turn, power-normalized agentic workload at a selected operating point — not a simple all-purpose throughput improvement. Nvidia also cited a combined Vera Rubin plus Groq 3 LPX claim of up to 35× higher token throughput per megawatt versus the older GB200 NVL72 for 2-trillion-plus-parameter long-context models. That’s yet another configuration and baseline, and it shouldn’t be blended into the 30× headline.
Both the GB300 and Vera Rubin software stacks are continuing to be optimized, meaning the generational ratio could shift as Nvidia, SemiAnalysis, and open-source serving-framework contributors update kernels, routing, and cache-management software. And throughput per megawatt is most valuable to operators constrained by electrical capacity. Buyers whose binding constraint is capital cost, rack space, availability, latency, software compatibility, or model quality may weight the result quite differently. The available Vera Rubin AgentX data Nvidia has cited covers one model. It doesn’t establish that the same gap applies to other major models, different model sizes, non-mixture-of-experts architectures, short-context chat, training, or customer-specific agent workloads. That’s where independent confirmation from SemiAnalysis will matter most — and until it arrives, the 30× remains a vendor claim, albeit one built on an open and reproducible benchmark framework.
