This article is the third installment in an AI model co-design series. It examines how speculative decoding can accelerate autoregressive LLM decoding by predicting multiple tokens per iteration and provides five practical guidelines for choosing draft length and draft mechanism across the throughput–interactivity Pareto frontier.
What speculative decoding does
Speculative decoding uses a small draft model to propose multiple next tokens (D) and a larger target model to verify them in one pass. The target model accepts proposed tokens sequentially until the first mismatch; decoding resumes from that position. Unless acceptance rules are relaxed, the final output equals that of standard decoding.
Key quantities:
- draft length (D): number of tokens proposed per target iteration
- acceptance length (AL): number of tokens accepted by the target per iteration; AL ranges from 1 to 1+D
The speedup is expressed as:
speedup = T_verif(B) * AL / (T_verif(B*(1+D)) + T_draft(B,D))
where B is batch size, T_verif(x) is verification time for x tokens, and T_draft(b,y) is draft generation time. Maximizing speedup requires choosing the optimal (D, AL, T_draft) tuple.
How to choose draft length
Ignoring draft latency, speculation speeds up decoding when
T_verif(B*(1+D)) / T_verif(B) < AL.
Verification compute scales with (1+D) while memory access does not, so increasing D until T_verif saturates (typically at the memory-bound to compute-bound transition) is the objective. The optimal D therefore depends on B and varies across the Pareto frontier.
Draft length and linear layers (GEMMs)
With speculation the GEMM-M for each target linear layer grows from M to M*(1+D). For a representative 6144x6144 GEMM, higher D values let GEMMs reach peak throughput at lower effective batch sizes: e.g., D=7 requires one-eighth of the batch needed at D=0 to become compute-bound. As MoE models get sparser and long-context workloads increase KV pressure, effective concurrency per expert falls, making larger D attractive.
Guideline 1: Increase speculative draft length to push GEMMs into the compute-bound region without increasing KV cache capacity pressure.
Draft length and attention performance
For reasoning and agentic workloads attention often dominates runtime in the throughput-oriented region. Decode attention has arithmetic intensity ≈ 2G, where G is the number of query heads sharing one KV head. Speculation raises this to 2G*(1+D) because speculated tokens reuse the same KV cache; effective attention GEMM-M becomes G*(1+D). On current GPUs attention kernels achieve good utilization at GEMM-M = 128, so a practical starting point when attention dominates is D = 128/G – 1.
Guideline 2: When attention dominates decode time, choose D = 128/G – 1 as a starting point.
Attention runtime also depends on tile size: runtime grows in steps as G*(1+D) crosses multiples of 128 (the software tile size for the benchmarked kernel). If G*(1+D) sits between two tile boundaries, the last tile is partially utilized but costs nearly as much as a full tile.
Guideline 3: If you pick D > 128/G – 1, prefer D values where G*(1+D) is a multiple of 128 to avoid tile underutilization.
The importance of Guideline 1 versus Guidelines 2–3 depends on how much runtime is spent in FFNs versus attention at the chosen operating point. Communication costs also grow with D, though compute–communication overlap can reduce overhead.
Draft length at very small batch sizes (far right of the Pareto curve)
When B is very small, fixed kernel setup and post-processing overhead dominate. These fixed costs don't rise much with the number of verified tokens, so verification overhead stays largely invariant with D. While MoEs may activate more experts as D increases, sharding strategies and efficient kernels (e.g., Grouped GEMMs) can keep overhead low. Larger D can therefore help in low-latency regimes provided acceptance stays high.
At very low latency the number of sequential kernel launches determines latency; kernel launches scale with layer count. For an autoregressive draft model with layer structure similar to the target, a simple approximation is:
speedup = (L_target * AL) / (L_target + D * L_draft)
Defining rho = L_draft / L_target gives speedup = AL / (1 + rho * D) and draft overhead O_d = rho * D. In short, D helps only while AL gains offset the draft overhead.
Guideline 4: At very low latency, increase D only while the AL gain justifies the added draft cost; prefer fast draft mechanisms.
Choosing a draft mechanism
After deciding how many tokens to speculate, choose how to generate them. Options differ in training, parameter, and runtime costs. Examples compared include:
- External draft model: small standalone LLM; higher serve-time memory and potentially high AL for larger D
- MTP: decoder layer(s) + linear projection attached to target; runs serially D times (higher O_d for large D)
- EAGLE-3: decoder layers + linear projection, target-attached post-training head
- DFlash and DSpark: fused or fused+Markov lightweight heads that can generate D tokens in one parallel step, lowering O_d
- Suffix / n-gram: model-free string matching, essentially O(1) lookup but lower AL, suited for repetitive workloads
A comparative table in the source lists inputs, generation method, training cost, serve-time memory and per-D speculation cost for these approaches.
Benchmarking AL versus D on SPEED-Bench (Qwen 3.5 122B A10B as target, 32K split) shows external drafts achieve higher AL at D>3, while DFlash/DSpark ALs level off earlier but benefit from smaller draft overhead due to parallel generation. For example, Qwen 3.5 35B A3B reaches AL=6 at D=9; a 4B external draft reaches AL>5. MTP produces drafts serially (O_d proportional to D), whereas a five-layer DFlash head can produce D tokens in one pass (lower O_d). For large targets the relative overheads shrink, but for small models at low latency DFlash/DSpark can be optimal despite lower AL.
Guideline 5: Choose the draft mechanism that maximizes decode speedup for your workload and hardware, balancing AL, draft latency, and training/deployment cost. Benchmark both AL and draft overhead under realistic serving conditions (the authors recommend SPEED-Bench and high-performance inference frameworks such as NVIDIA TensorRT LLM).
Training and maintenance considerations
Drafts differ in training scope and cost. MTP must be co-trained or realigned on finetuning; EAGLE-3, DFlash, DSpark can be added as post-training heads. External drafts can be trained from scratch, distilled from the target, or adapted via cross-model techniques; the latter reduces training cost at the expense of AL. If the target model is finetuned, remeasure AL on representative workloads because target-attached drafters may lose acceptance and require retraining or adaptation.
Quick checklist (the five guidelines)
- Increase draft length to push GEMMs into the compute-bound region without increasing KV cache pressure.
- If attention dominates, start with D = 128/G – 1.
- For larger D, prefer values where G*(1+D) is a multiple of 128 to match attention tile size.
- At very low latency, prefer fast draft mechanisms and raise D only while AL gains cover draft costs.
- Select the draft mechanism that yields the best decode speedup for your workload and hardware; benchmark AL and draft overhead with realistic prompts and tasks.
The post includes pointers to NVIDIA Model-Optimizer examples for EAGLE-3, DFlash, and DSpark, and references to fine-tuning DSpark and quantization to FP8 or NVFP4 for NVIDIA Nemotron 3.5 Lightning as practical starting points. Acknowledgments list the NVIDIA contributors to the work.
