NVIDIA Snapshot is an open-source component of the NVIDIA DSX AI Factory Platform that provides checkpoint-and-restore for GPU workloads running on Kubernetes. Its purpose is to make fully initialized GPU inference workloads reusable across replicas so scaling and recovery happen in seconds rather than minutes.
Why this matters
Cold starts for large language model (LLM) inference—loading weights, compiling kernels, allocating KV caches, and capturing CUDA graphs—can take minutes. When traffic spikes or a replica fails, available GPUs do not immediately translate to serving capacity. Snapshot captures the fully initialized state of a workload as a portable checkpoint and restores that state on compatible nodes to bring replicas online in seconds.
How it works (summary)
Snapshot provides two primary operations for GPU Pods:
- Checkpoint: capture CPU and GPU memory of a fully initialized workload as a portable artifact (a one-time cost per model/configuration).
- Restore: start a new Pod from that artifact on any node with the same GPU model and driver; the workload continues where it was checkpointed, skipping repeated steps like weight loading, compilation, and graph capture.
These operations are exposed through Kubernetes custom resources. To checkpoint an already running Pod, create a PodSnapshot that names the Pod. To create and checkpoint a Pod as soon as it is ready, use a SnapshotJob. To restore, annotate a new Pod with nvidia.com/restore-from naming the PodSnapshot. The workload needs only a brief pause before checkpoint and a brief handshake to resume after restore. The repository includes ready-made examples for vLLM, SGLang, and NVIDIA TensorRT LLM.
Multi-GPU and MoE support
Snapshot can checkpoint and restore workloads that span multiple GPUs on a single node as a single artifact across ranks. This supports fast startup for large mixture-of-experts (MoE) models using tensor parallelism (TP), data and expert parallelism (DEP), and decode context parallelism (DCP). Workloads must meet Snapshot’s checkpoint/restore and runtime compatibility requirements; example configurations are provided in the repo.
Benchmarks — hardware and environment
Benchmarks were run on NVIDIA B200 GPUs with NVIDIA driver 615.71.09 and VAST (NFS) storage. The inference engines and versions tested were:
- vLLM 0.31.0
- SGLang 0.5.21
- TensorRT LLM 1.3.0
Tested models ranged from 600M-parameter models up to MoE checkpoints of roughly 600 GB–1 TB across eight GPUs.
Key findings
The measurements produced four main findings:
- Restore times under 43 seconds: Snapshot restored replicas in 3.6–42.7 seconds across all tested configurations, yielding 8x–36x speedups versus cold starts.
- Restore skips more than weight loading: by restoring an initialized process image, Snapshot avoids repeated weight loading, kernel compilation/autotuning, and CUDA graph capture.
- Checkpointing pays off immediately: for every tested configuration, checkpoint time plus a subsequent restore was shorter than a single cold start. Checkpoint creation took 15 seconds to 5.5 minutes depending on the model and setup.
- Moving GPU state dominates large restores: for eight-GPU MoE setups, copying GPU state back into GPU memory took 18–29 seconds; CRIU-based CPU process restores took 4.2–10.8 seconds.
Representative results
The published tables include checkpoint size and time, cold-start breakdown (startup, model load, initialization, compilation/autotuning/graph capture) and Snapshot restore stages (agent setup, CRIU restore, CUDA restore, wake/remap). A few examples from the data:
- Qwen3-0.6B (vLLM, TP1): checkpoint size 8.13 GB, checkpoint time 15.30 s, cold start 104.43 s, Snapshot restore 4.66 s (22.40x speedup).
- GLM-5.3-NVFP4 (vLLM, TEP8, 8 GPUs): checkpoint size 604.00 GB, checkpoint time 282.80 s, cold start 393.65 s, Snapshot restore 28.64 s (13.74x speedup).
- DeepSeek-V4.1-Flash (vLLM, TEP8, 8 GPUs): checkpoint size 647.18 GB, checkpoint time 274.75 s, cold start 1110.65 s, Snapshot restore 30.96 s (35.87x speedup).
Across configurations, Snapshot restored replicas in under 45 seconds while cold starts ranged from roughly one to many minutes depending on model and parallelism.
Methodology and limitations
Each configuration was measured using two strategies on the same hardware and software stack:
- Cold start: measured from process launch until engine initialization returns (divided into startup, model load, initialization, compilation/autotuning/graph capture).
- Snapshot: measured from the moment Snapshot is asked to restore until the workload is serving (divided into agent setup, CRIU restore, CUDA restore, wake/remap).
Both measurements exclude pulling container images and container runtime setup. The KV cache is released before every checkpoint and is not part of the checkpoint artifact. Restore performance depends on storage backend and network bandwidth because CRIU restore and GPU state transfer are I/O intensive.
Try it out
Installation and usage guides are available in the NVIDIA Snapshot repository, which includes benchmarks and reproduction instructions. There are user guides for using Snapshot with NVIDIA Dynamo to restore workers. For integrations or feedback, open an issue in the repository or join the NVIDIA Dynamo Slack workspace and the #snapshot-dev channel. NVIDIA is also demonstrating Snapshot at booth #170 at KubeCon.
Summary
NVIDIA Snapshot enables portable checkpointing and rapid restore of fully initialized GPU inference workloads on Kubernetes. Benchmarks on NVIDIA B200 hardware show restores in 3.6–42.7 seconds—substantially faster than cold starts—and the checkpoint-plus-restore workflow usually pays off compared to a single cold start, especially for large MoE and multi-GPU deployments.
