When an agentic system routes work between models, the receiving model normally must re‑run the entire prefill stage that computes keys and values for all past tokens and populates the KV cache. That recomputation drives up latency and compute costs, creating a bottleneck for long‑horizon, multi‑LLM workflows.
Researchers at Nvidia have introduced a cross‑model KV cache transfer technique that directly maps a source model’s prefilled KV cache into the target model’s expected format. The approach uses simple linear math rather than heavy neural networks and is intended for real‑world agentic applications where context accumulates over many turns.
Why swapping models is expensive
Large language models operate in two phases: a prefill (an initial forward pass that fills the KV cache with keys and values for each input token) and a decode (generating tokens while reading from that cache). The prefill cost scales with both model size and input length, so long sessions become expensive. If the KV cache is invalidated — as happens when switching to a model with a different architecture — the receiving model must pay the full prefill cost again.
Model switches occur in practical scenarios: routing a complex reasoning step to a larger model, or stepping down to a smaller model to save resources. Because different LLMs expect cache inputs in different formats, model swapping has required full recomputation until now.
Mapping memory between models without re‑prefill
Nvidia’s research explores how to transform a source model’s KV cache into the format a target model expects, without rerunning prefill. Successful cross‑model transfer enables two common workflows:
- Small→Large: a cheap model handles routine turns; when complex reasoning is needed, the KV cache is mapped to a larger model to continue processing.
- Large→Small: a large model does the heavy initial work (e.g., unpacking a big system prompt or dense document), then the resulting KV is mapped down to a smaller model for subsequent interactive turns.
Previous attempts at KV transfer often required expensive gradient‑based training or strict architectural constraints. This study focuses on within‑family transfers (e.g., different sizes in the Qwen, Llama, or Ministral families), where tokenizers and core architectural patterns are shared but model depth and size differ.
Key insight and the closed‑form mapper
The central finding is that cross‑model KV cache exhibits a strongly linear structure, so mapping can be done with linear algebra. For example, transferring KV from Qwen3 14B to 32B, a single‑layer linear regression recovered 56% of the variance in the target’s keys and 32% in values; combining multiple source layers raised those figures to 79% and 65%.
To make this practical, the researchers designed a closed‑form, per‑head ridge mapper composed of three parts:
- Per‑head ridge regression: fit a simple linear regression independently for every attention head using a small calibration set of a few hundred text sequences rather than training a large neural network.
- Cross‑layer source selection: because source and target have different layer counts, the mapper picks the most predictive source layers for each target layer so it uses only the most useful memory pieces.
- Content‑space mapping (RoPE removal): the mapper strips Rotary Position Embedding (RoPE) encodings before translation so the mapping can generalize to sequences longer than the calibration data.
The calibration set used to fit the mapper contained just 500 sequences of 1,024 tokens each.
Experimental evaluation: accuracy and speed
The team evaluated the pipeline across six matched‑KV model families (source and target share KV head counts and per‑head dimensions), including Qwen3, Llama 3.1, and Ministral 3. Sizes ranged from 3B to 70B parameters, including an 8.8× jump from Llama 3.1 8B to 70B. They tested on five accuracy benchmarks (ARC‑Challenge, HellaSwag, WinoGrande, MMLU, GSM8K), WikiText‑2 perplexity, and a multi‑turn conversation task (CoQA).
Key results:
- For four of six tested pairs, the closed‑form linear ridge mapper preserved 73%–98% of the target model’s standalone prefill accuracy. The Llama 3.1 8B→70B jump retained 72.8% of target accuracy.
- The transfer ran 2.7× to 25× faster than full re‑prefilling. For example, converting a 32,768‑token KV cache from Qwen3 14B to 32B took 278 milliseconds versus nearly 7 seconds for a traditional re‑prefill.
- On multi‑turn conversation tests, the accuracy drift between the transferred cache and the target baseline remained very small across ten turns, indicating stability for long agentic sessions.
Limitations and remedies
The linear method struggled on some pairs: two Ministral configurations showed sharp degradation because the linear fit failed to extrapolate beyond the calibration distribution. The researchers addressed that by replacing the linear mapper with a nonlinear multi‑layer perceptron (MLP) with two 1,024‑unit hidden layers trained on the same calibration data. That approach recovered accuracy above 90% at the cost of added training and complexity.
The present study is limited to within‑family, matched‑KV transfers, but the authors suggest the framework could be extended to cross‑family cases, mismatched KV head counts, or hybrid architectures in future work.
Context within broader industry efforts
Cross‑model KV transfer is one of several recent approaches aimed at the KV‑cache bottleneck as enterprises scale LLMs for long and complex tasks. Other examples include:
- Nvidia’s Dynamic Memory Sparsification (DMS), which evicts less important tokens to reduce reasoning costs by up to 8×.
- MIT’s Attention Matching, an algebraic compaction that can compress KV caches by large factors (reported up to ~50×) without quality loss.
- Nvidia’s KV Cache Transform Coding (KVTC), which applies media‑compression concepts to shrink memory footprints (~20×) without modifying model weights.
- Optimizers like IndexCache that remove redundant layer computations for faster time‑to‑first‑token, and architecture changes in models like DeepSeek and the GLM series that optimize KV handling at the model level.
As models handle longer contexts and more complex pipelines, memory infrastructure becomes as important as model capacity. Cross‑model KV cache transfer gives developers another practical tool to reduce inference cost and latency in multi‑model agentic systems.
