NVIDIA positions Groq 3 LPX as the interactive AI inference accelerator for the NVIDIA Vera Rubin platform. The platform’s core is the NVIDIA Vera Rubin NVL72, described by NVIDIA as their most versatile machine, offering high throughput and interactivity across a broad range of AI workloads — from small to large models, open or closed. When paired with Vera Rubin NVL72, Groq 3 LPX extends the platform’s capability to support the highest‑interactivity serving tiers and to power AI factory‑style user experiences.
Why long context at high interactivity matters
Agentic, multi‑turn sessions continually append each turn’s output to a growing context that feeds subsequent turns. Across lengthy sessions the context size can rise to hundreds of thousands of tokens. Without support for long context, agents can only consider a fraction of prior relevant information, which limits their capabilities regardless of raw model quality or speed. The most capable agents therefore must sustain both low latency and retention of long context as sessions grow.
Why ultrafast interactivity with long context is challenging
Serving a model at more than 3,000 output tokens per second per user while managing a KV cache for 100K input tokens creates a specific systems challenge. Techniques such as tensor parallelism (TP) can deliver order‑of‑magnitude speedups, but they depend on efficient coordination of many small collective operations. Highest interactivity requires very small batch sizes where fixed coordination costs can negate TP’s benefits.
Each chip‑to‑chip transfer has two timing components: first‑bit latency (setting up which link to use, synchronizing sender and receiver, arbitrating conflicts) and transfer time (moving the bytes across the network relative to bandwidth). For small transfers, first‑bit latency often becomes the bottleneck, so reducing it is critical to keep TP effective at small batch sizes.
How Groq 3 LPX addresses these costs
Groq 3 LPX reduces coordination overhead with compiler‑scheduled workload planning, tight chip‑to‑chip (C2C) networking within the rack, and heavy overlap of compute with interprocessor communication.
-
Deterministic, preplanned interchip communication: Groq 3 LPX uses a deterministic execution model. The compiler knows the individual compute units across 256 LP30 local processing units (LPUs), the total 128 GB of SRAM across those chips, and the 96 C2C links per chip running at 112 Gbps. Using that visibility, the compiler produces an exact, cycle‑level schedule for how the workload runs before it begins, which removes the need for run‑time arbitration of transfers.
-
Point‑to‑point links and LPU routing: Links are point‑to‑point between pairs of LPUs and LPUs can act as routers as well as processors, enabling routed transfers through other LPUs when needed.
-
Fine‑grained compute‑communication overlap: The compiler schedules work at the granularity of 320‑byte vectors. Because matrix multiplications can be expressed as a series of dot products, LPUs can compute slices (e.g., 320 columns) of an output and start sending those results immediately. This allows data transfer to begin as soon as enough of a result is ready to fill a 320‑byte vector instead of waiting for an entire matrix operation to finish, increasing overlap and reducing communication tails, which matters for small tensors.
Together, these design choices drive down first‑bit latency and allow tensor parallelism to provide speedups even in the small‑batch, high‑interactivity regime.
Third‑party benchmark: Artificial Analysis 100K test
Artificial Analysis, which distributes standard benchmarking suites for serving speed at both 10K and 100K context lengths, ran the Gemma 4 model (31B parameters, released April 2026) on a Groq 3 LPX system hosted in NVIDIA data centers. For the 100K context benchmark they measured a median output speed of 3,431 tokens per second. Using the same suite at 10K context length, the median speed was 3,382 tokens per second. Both tests used the common "o200K_basetokens" tokenizer.
NVIDIA and Artificial Analysis verified that the configuration produced no loss of precision or model quality in outputs.
Coding benchmark: SPEED‑Bench
On the open‑source SPEED‑Bench coding benchmark with the same Gemma 4 model, the Groq 3 LPX system achieved a median output speed of 4,767 tokens per second and a P80 of 5,520 tokens per second — meaning 20% of tasks finished at over 5,500 tokens per second.
Practical impact
At the measured speeds, decoding 5,000 tokens takes roughly 1.5 seconds; by comparison, at 100 tokens per second the same decoding would take about 50 seconds. The article notes that many current agentic tools operate closer to ~60 tokens per second. This performance gap can materially improve user experience for agentic coding and other multi‑turn workflows where agents read hundreds of files or otherwise create contexts exceeding 100K tokens.
How Groq 3 LPX and Vera Rubin NVL72 can be combined
Pairing Groq 3 LPX’s low‑latency deterministic execution with Vera Rubin NVL72 racks supports multiple serving configurations, including:
- Standard prefill‑decode disaggregation: NVL72 performs prefill and hands off the KV cache per turn; Groq 3 LPX performs the decode using the KV cache and weights in SRAM.
- Attention‑FFN disaggregation: NVL72 computes attention and holds KV cache in DRAM while Groq 3 LPX executes FFN layers; only intermediate tokens cross racks per full‑attention layer.
- External‑drafter speculative decoding: Groq 3 LPX runs a small draft model ahead of the large target model on NVL72; NVL72 verifies and commits tokens and returns rejected positions for the next chunk. Each rack keeps its own KV cache; only draft tokens cross links.
These disaggregation options let each rack focus on the work it executes best and together push the Vera Rubin platform toward higher interactivity and lower latency.
Conclusion
A third‑party benchmark confirms that Groq 3 LPX delivers leading interactivity at context lengths relevant for agentic workloads. Its compiler‑driven, deterministic scheduling and fine‑grained overlap of compute and communication enable tensor‑parallel speedups even at small batch sizes. NVIDIA has also published further technical details for readers who want a deeper dive into Groq 3 LPX’s design.
