Over the past two years many companies, including banks, purchased private GPU clusters or reserved cloud nodes for reasons such as data sovereignty and avoidance of vendor lock-in. Despite this, utilization often remains low: for example, a bank in a Singapore data center pays for eight NVIDIA H100s that spend much of the night idle. This phenomenon has been called the “GPU hangover.”
Why utilization is low: the memory wall and lack of concurrency
Standard autoregressive models (for example from the Llama, Mistral, or GPT families) generate tokens one at a time. The model weights remain in the GPU’s high-bandwidth memory, but the arithmetic happens in a small on-chip buffer that cannot hold a multi-billion-parameter model. As a result, for every single token the full set of weights must be streamed through the compute units again. The system is limited by memory bandwidth, not raw compute. Measured as arithmetic intensity (operations per byte moved), batch size one puts autoregressive decoding near 1, whereas modern GPUs are designed for intensities in the hundreds. The chip is therefore starved: compute units are idle between memory transfers.
The usual escape is batching: read the weights once and apply them to many requests’ next tokens at the same time, amortizing the expensive read over much more useful work. This is why large public APIs, which run enormous batches across thousands of users, achieve high throughput and low per-token cost.
Which workloads are not batchable? Real-time, latency-sensitive requests
The key question is whether you can accumulate concurrent work. An overnight offline queue of millions of documents (for KYC, say) is trivially batchable because nobody waits. By contrast, a developer’s code completion, a bank onboarding check at the counter, or a single agentic workflow step often requires a subsecond response and cannot be delayed to form a large batch. Autoregressive decoding in single-stream mode becomes memory-bound: it may only emit a few dozen tokens per second, so producing several hundred tokens can take multiple seconds even with speculative techniques.
Where text diffusion helps
Text-diffusion decoding borrows the image-modeling mechanism: it begins with a masked or noisy block of tokens and refines the whole block in parallel across a few denoising passes. Instead of behaving like a typewriter, it acts like an editor revising an entire draft at once. Because each pass performs real arithmetic across the whole block, diffusion is compute-bound even at batch size one. If autoregressive intensity sits near 1, a comparable diffusion model’s intensity lands in the hundreds — it can saturate the GPU without needing concurrency.
Practical figures reported in the field back this up: Inception Labs’ Mercury reported over 1,100 tokens per second on H100s for code generation, and the Mercury 2 release (2026) reported roughly 1,000 tokens per second on Blackwell at low latency. Google demonstrated the approach at frontier scale with Gemini Diffusion, and open-source LLaDA work shows diffusion models follow autoregressive-like scaling laws. These are early but real results: Mercury 2 is commercially available, Gemini Diffusion is in enterprise preview with general availability expected later in 2026, and open models are maturing rapidly.
In short: for strict single-stream latency targets, diffusion can be an order of magnitude faster because autoregressive models are trapped by the memory wall and cannot be batched out of it.
But don’t forget autoregressive optimizations
A fair comparison must account for what autoregressive serving already achieves. Speculative decoding and its descendants (Medusa and EAGLE) use a small draft model to propose multiple tokens that the main model verifies in a single pass, typically giving roughly 2–4× single-stream speedups without quality loss. Mixture-of-experts models reduce memory movement by activating only a fraction of weights per token. Thus the real question is not “autoregressive vs. diffusion” in the abstract, but whether diffusion’s structural parallelism beats the incremental gains available from speculative decoding or MoE for the workload at hand. For tight single-stream latency diffusion’s edge is large; for offline batch neither trick matters much because large batches already render both architectures compute-bound.
Economics: nodes, throughput, utilization, and when owned compute wins
Economics collapse to one identity:
Effective cost per token = node cost per hour ÷ (throughput × utilization)
Public APIs are priced per token and don’t charge idle-node overhead, while owned compute costs per hour so per-token cost depends on throughput and utilization. An AWS p5.48xlarge (eight H100s) lists near $55 per hour on demand; one-year Savings Plans cut that by roughly 40% to about $33 per hour. Against a cheap commodity API that serves a small model for under $1 per million tokens, owned compute loses on pure cost regardless of architecture. Diffusion’s economic advantage appears only in two situations: (1) when the token you’d otherwise buy is expensive — frontier reasoning outputs priced at, say, $5–$15 per million tokens — and a saturated owned node undercuts the API; or (2) when the data cannot be sent to an external API at all for sovereignty/regulatory reasons, so the comparison is owned diffusion versus owned autoregressive. Most regulated enterprises fall into the latter case.
Bank example: offline batch vs. real-time parsing
The bank’s document pipeline illustrates the distinction:
-
Overnight batch: millions of KYC packets, letters of credit, loan files converted to JSON while no one waits. This workload is trivially batchable. A standard autoregressive model with continuous batching can run at several thousand tokens per second and clear the queue on a single node. Diffusion may be somewhat faster, but both fit one box at similar cost.
-
Real-time path: a relationship manager needs documents parsed within a second while the customer waits; an officer needs clearance immediately. These requests arrive one at a time with hard latency budgets. Autoregressive single-stream decode emits only tens of tokens per second, so outputs taking a few hundred tokens require several seconds. Speculative decoding improves speed but often does not reach interactive targets. Diffusion returns the same record well under a second. The cost difference shows up in node count: to meet subsecond targets with autoregressive models you must keep batches tiny and therefore overprovision many nodes for peak; diffusion clears each request quickly so far fewer nodes are needed to meet the same service level. The savings are real, and they stem from the inability to batch under the latency constraint.
Constraints and trade-offs
Diffusion carries real limitations:
-
Quality: diffusion models typically land around 85–95% of strong autoregressive baselines on reported measures. They are competitive on structured output and field extraction but can trail by 5–15 percentage points on hard reasoning tasks. Those vendor and secondary figures should be independently verified on your own data. For low-stakes extraction this may be acceptable; for credit decisions it is not — serious deployments must budget for fallbacks when confidence is low and roll that cost into the effective per-token rate.
-
Compute cost: being compute-bound is itself a cost. Diffusion attains high intensity partly by doing more total arithmetic per useful token, so the relevant metric is tokens-per-dollar at an acceptable quality threshold, not utilization alone.
-
Tooling and maturity: diffusion serving in 2026 is young. Open-source diffusion stacks are roughly where autoregressive serving was in early 2024: functional and improving but lacking the mature inference stacks teams expect from vLLM or TensorRT-LLM.
-
Moving baseline: speculative decoding, better schedulers, and MoE models are narrowing the gap without a wholesale model swap. Diffusion must beat a moving target, not a naive static baseline.
Practical rule: route by batchability and token value
A practical routing rule sharper than “customer-facing on API, internal on owned compute” uses two axes:
- Can the work be batched (offline-tolerant) or is it latency-bound and serial?
- What is the value per token (low-cost generation vs. high-value reasoning)?
- Latency-bound, decode-heavy, low-to-moderate-value generation (code completion, real-time extraction, agentic workflow chatter) is the diffusion sweet spot: batching is unavailable, quality tradeoffs are tolerable, and a fast owned node beats both an overprovisioned autoregressive fleet and an expensive API.
- High-value reasoning with costly errors should stay on frontier autoregressive models.
- Offline batch of any value density should use whatever the organization already runs well, because batching makes it efficient.
Measure tokens-per-dollar at your acceptable quality bar and repeat the analysis as API and hardware prices change.
Conclusion
The hangover is not that enterprises bought the wrong hardware: many bought it for sovereignty, data control, or to avoid lock-in, valid reasons that won’t disappear. The issue is expecting that owned GPUs would behave like public cloud economics when the most valuable internal workloads — latency-bound, single-stream requests — can never reach the concurrency that cloud providers exploit. Text diffusion is not a universal way to beat the API nor a blanket upgrade for every enterprise workload. It is a precise tool for a precise gap: latency-bound, decode-heavy, sovereignty-constrained work where batching is impossible and an autoregressive model leaves a node both starved and overprovisioned. For copilots, real-time checks, and agentic steps that must answer now, diffusion can turn a guilty line item into a saturated asset on a fraction of the boxes an autoregressive alternative would need. That is a narrower but more durable claim than promising to rescue overall hardware ROI. The right architecture on the right hardware, carrying the right tokens — and knowing which tokens those are — remains the core of enterprise AI procurement.
Further reading
The article references Inception Labs’ “Mercury” and Mercury 2 materials (2026), Gemini Diffusion, LLaDA research, work on speculative decoding (Leviathan et al., Medusa, EAGLE), and AWS P5 pricing publications. Reported throughput and accuracy figures are engineering approximations for roughly 70B-class models; substitute your own measured numbers, batch sizes, and sequence lengths before any procurement decision.
