Agentic AI tasks, long documents and large codebases increasingly require models to handle contexts containing hundreds of thousands of tokens. At that scale, attention becomes particularly expensive in transformers: conventional dense attention compares every query with every key in the context. With a 128,000-token input, the operation can account for much of the prefill time in each layer.
Sparse attention instead focuses each query on a selected part of the context, reducing computation and memory traffic. The saving does not eliminate work across the full context, however: an indexer must still find the relevant keys. In the measured DeepSeek V3.2 model, indexing and Top-k selection together accounted for 47% of per-layer prefill time at 128,000 tokens.
The analysis focuses on sparsity trained into the model, rather than methods applied after training, such as BLASST, MInference or Quest. Its measurements examine how indexers and attention operations can be designed for NVIDIA graphics processing units.
Three stages with different costs
In the first stage of sparse attention, the indexer assigns scores to the context's keys. Top-k then retains the k highest-scoring entries, and core attention operates only on the selected keys and values. The cost of the latter two stages depends on the number selected, while the indexer's cost still grows with the full context length.
Recent approaches make different trade-offs. DeepSeek Sparse Attention, for example, uses 64 query heads, one key head and a head dimension of 128, selecting individual tokens. GLM 5.2 uses 32 query heads and shares its index across layers; GLM 5.3 Flash works with blocks of four. Qwen QSA uses four query heads and blocks of four, while MiniMax MSA pools scores into blocks of 128 tokens. Other approaches, including LongCat LSA-Flash, use hierarchical indexing or share indexes across layers.
The performance measurements used a production FP8 indexer kernel. Unless otherwise stated, they used 64,000 indexer keys, a compression ratio of four, a head dimension of 128 and one key head. The prefill batch size was one and the decode batch size was 128.
Heads and speculative decoding
During prefill, the indexer scores many queries at once, so performance is generally limited by computation. During decoding, it evaluates only a few queries per step while reading the entire key cache, making memory traffic the main bottleneck.
As a result, additional query heads affect the two phases differently. In the measurements, a 16-fold increase in group size changed decode runtime by only 2%, because the same key cache had to be read. During prefill, a larger group meant more computation; above a group size of roughly 16, runtime grew with the group size.
Speculative decoding can also make use of the same cache read: the model can verify D proposed draft tokens and one new token together. This remains relatively inexpensive while the query rows fit in a single Tensor Core tile. With an FP32 epilogue, the practical limit is about 200 rows: six query tokens at a group size of 32, or 26 at a group size of eight. Within those ranges, work increased sixfold and 26-fold respectively, while runtime rose only 1.37-fold and 1.87-fold.
An FP16 epilogue can expand the tile to 256 rows and delivered a 1.31-fold speedup in the tested configuration with a group size of 64 and four query tokens per step. Using FP16 accumulation as well produced a 1.45-fold speedup, but only when the matrix multiplication outputs remain within the FP16 range.
Increasing the number of key heads, by contrast, requires additional cache reads during decoding. Moving from one key head to eight increased decode runtime 7.6-fold, compared with 1.3–1.6-fold during prefill. The analysis therefore recommends one key head and a reduction across the query-head scores.
Head dimension, compression and Top-k
Increasing the head dimension means more computation and larger keys, but the prefill effect is not always proportional. Raising the dimension from 64 to 128 took only 1.1–1.2 times as long in the measurements, because at the smaller dimension the Tensor Cores were waiting for the epilogue. During decoding, however, reading larger keys increased cost throughout. The recommended dimensions are 128 or 256, with most of the examined designs choosing 128.
Compressing the indexer cache directly reduces the number of keys to score. At a compression ratio of four, one indexer key represents four tokens. In the measurements, halving the compression ratio generally doubled runtime in both phases. The number of keys can also be reduced through hierarchical indexing or by sharing an index across layers. Indexer compression is separate from compression of core attention's own key-value cache.
Top-k cost is likewise driven mainly by the number of keys examined. Doubling that number took about 1.7 times as long in prefill and 1.5 times as long in decode. By contrast, increasing the number of retained entries 64-fold raised Top-k time by less than twice in prefill and by 1.7 times in decode.
To reduce the cost further, the Guess-Verify-Refine Top-k method in NVIDIA TensorRT LLM starts with the previous decode step's result and then verifies it. During decoding on DeepSeek V3.2 and DeepSeek V4, it was 2.1–2.4 times faster than a standard radix-based selection. Another approach fuses Top-k and the indexer into one kernel, avoiding a write and subsequent read of the full score matrix; the open-source FlashInfer implementation provides an example.
Why blocks matter
With token-level selection, the required keys and values may be scattered across the cache, forcing the kernel to handle separate addresses and fetches. Block-level selection allows adjacent entries to be loaded together, and the same block can be reused by multiple queries.
The analysis compared block sizes with 2,048 selected keys per query. Although the amount of computation and data fetched remained the same, halving the block size nearly doubled runtime in both prefill and decode because more partial operations and result merges were needed. The analysis therefore recommends larger blocks and, in core attention, grouping many query heads around each shared key-value head.
The central finding is that sparse attention delivers meaningful benefits for long contexts when indexing and selection, as well as the final attention operation, are designed around the hardware. Fewer scored keys, fused operations and efficient data reuse can together reduce costs.



