NVIDIA Dynamo’s shadow engine speeds LLM worker recovery from minutes to seconds

NVIDIA Dynamo’s preview capability called shadow engine recovery keeps a fully initialized standby engine on the same GPUs as an active engine and uses a GPU Memory Service (GMS) to share weight allocations without duplicating them in HBM. When an active engine process fails, the shadow can be promoted and resume serving within seconds while the failed engine reinitializes in the background.

Why standard LLM recovery takes minutes

Typical recovery after an LLM engine process failure involves a cold restart: loading weights from storage into HBM, compiling kernels, and recapturing CUDA graphs. For large models this initialization can take minutes, and during that time surviving workers must absorb all traffic. Two fundamental obstacles prevent a fresh engine from skipping that cost:

• Weights are tied to the engine process. GPU memory is associated with the engine’s CUDA context; when the process exits the driver frees its HBM allocations, forcing a replacement process to reload weights.
• Some initialization state is non-transferable. NCCL and torch.distributed communicators and CUDA graphs are bound to the running process and its virtual addresses, so they must be recreated on restart.

The shadow engine approach

Shadow engine recovery resolves these issues by decoupling weight lifetime from engine processes and doing non-transferable initialization ahead of time.

GPU Memory Service (GMS)

GMS is a per-GPU sidecar that owns physical GPU memory regions (such as weights) independently of engine processes. It allocates physical pages, hands out handles, and arbitrates read/write ownership. Engines connect, import handles, and map the underlying pages to virtual addresses in their own CUDA contexts. Mapping occurs at startup; GMS is not on the data path for subsequent accesses. This uses the CUDA Virtual Memory Management API so that physical allocations and virtual addresses have independent lifetimes. Physical allocations are reference-counted and therefore survive as long as any mapping exists. Two engines mapping the same tensor access the same physical bytes via context-local virtual addresses, and kernel reads cost no more than if the engine had allocated the memory itself. The practical consequences are:

• Weights persist across engine failures (the GMS reference keeps physical pages resident).
• Weights can be shared between concurrent engines, so a second engine on the same GPU has zero marginal weight cost. Integration into inference frameworks is a narrow change: vLLM, SGLang, and NVIDIA TensorRT-LLM integrate GMS through a custom torch.cuda.CUDAPluggableAllocator bound to the weight pool; from the engine’s perspective weights remain ordinary torch.Tensor objects. GMS currently does not support the KV cache in preview, but that capability is under active development.

Shadow engines: preinitialized standbys

A shadow engine is a fully initialized engine process that remains idle on the same GPUs as the active engine. Thanks to GMS weight sharing, the shadow does not need a separate HBM copy of weights, making it feasible to co-reside two engines on the same devices. A shadow completes the same startup steps as an active engine: connect to local GMS and import weight mappings, establish communicators (NCCL and NIXL), capture CUDA graphs, and perform warm-up. Once ready it parks: it releases materializable parts of memory (notably it does not materialize the KV cache) and blocks, waiting on a lock. Precomputed on the parked shadow:

• CUDA context, captured graphs, and communicators (the non-transferable state).
• GMS weight mappings (handles already imported).
  Deferred work:
• KV cache materialization: the shadow reserves an address range but only materializes the physical cache when promoted. A parked shadow therefore keeps a small footprint (context, graphs, communicators, and handles) and no duplicate weights or KV cache, enabling fast promotion.

The worker unit and leader election

Each worker pod contains two engine containers, a GMS sidecar, and a shared lock to elect the active engine. At steady state one engine holds the lock, is awake, connected to GMS, holds a materialized KV cache, and is registered with the frontend router. The other is initialized and connected to GMS but dormant and blocked on the lock. The lock is implemented using a POSIX flock on a shared file so that when the active process exits (normal shutdown, segfault, or SIGKILL) the kernel reaps file descriptors and the shadow can acquire the lock and begin serving. This pattern makes each engine’s startup a short leader election.

Recovery sequence in detail

A worker goes through four phases:

• T0 Steady: Engine A active, Engine B parked.
• T1 Failure: Engine A’s process exits; the worker is briefly unroutable until the shadow registers.
• T2 Cutover: Engine B acquires the lock, wakes, remaps weights via GMS, materializes KV cache, and re-registers with the router; Engine A is restarted by the orchestrator.
• T3 Restarted: Engine A finishes initialization and enters the shadow state; steady state resumes with roles swapped. Because the shadow has precomputed context, graphs, communicators, and weight mappings, the critical path only includes acquiring the lock, remapping weights, and materializing the KV cache, which completes in seconds.

Memory accounting

• Weights: allocated once by GMS and mapped read-only by every engine; not duplicated.
• KV cache: materialized only by the active engine; released on engine death so the shadow can materialize it.
• Buffers and graphs: NCCL buffers, CUDA contexts, and captured graphs are held by each engine (including parked shadows) and represent the standing cost of a parked shadow.

Benchmarks: GLM-5.2 two-worker experiment

To measure the impact, the team compared shadow engine recovery with a cold restart after killing one worker in a two-worker fleet. Setup: two workers serving GLM-5.2 quantized to NVFP4 on NVIDIA B200 nodes (one worker per node), TP=8, 200K max context, FP8 KV cache. A single frontend round-robin distributed traffic. The synthetic load used 32,000 input tokens and 1,000 output tokens per request at 0.7 requests/sec. Both arms ran identical engine builds; the only difference was whether the shadow engine was enabled. The fault was a SIGKILL to one worker after steady state, followed by 600 s of observation. Results (window after the fault):

• Time until a second worker serves again: cold restart 283 s vs shadow recovery 7.3 s (1.7 s detection + 5.6 s promotion).
• TTFT p50 after the fault: cold restart 23,815 ms vs shadow 1,311 ms.
• Decode rate p50 after the fault: cold restart 12 tok/s/user vs shadow 46 tok/s/user.
• Requests with >5 s to first token: cold restart 201 of 399 vs shadow 1 of 398. The shadow approach reduced recovery time by nearly 39× and materially improved time-to-first-token and decode rate, avoiding most SLA violations observed in the baseline.

Current scope, limitations and roadmap

Shadow engine recovery is in preview and has deployment requirements and limitations:

• It addresses common engine process failures but does not handle hardware, node, or multi-node failures (those still use standard rescheduling).
• It requires Kubernetes with Dynamic Resource Allocation (DRA): Kubernetes 1.34+ with DRA enabled and the NVIDIA GPU DRA driver installed.
• The preview does not yet support sharing the KV cache via GMS; carrying cache state across a promotion (both prefix-cache index and cache memory) is active work to reduce the post-cutover TTFT bump.
• vLLM is the primary supported backend today. The team is working to stabilize the implementation, broaden workload support, and roll the feature out incrementally over coming months. Dynamo Snapshot can be combined with the recovery feature to reduce contention during shadow initialization.

Try it and contribute

To try Shadow Engine Recovery, start with the Kubernetes quickstart to create a running deployment, follow the Shadow Engine Recovery deployment workflow, and use the vLLM failover example as a complete manifest. The ai-dynamo/dynamo repository is the place to ask questions, report issues, or contribute.

Why this matters

By decoupling weight lifetime from engine processes and preinitializing standbys, shadow engine recovery significantly shortens the window of degraded service after an engine process failure—from minutes to seconds in the tested scenario—improving resilience for latency-sensitive LLM inference deployments.