NVIDIA ModelExpress (MX) reduces the repeated time and bandwidth cost of moving large model checkpoints by first checking whether a compatible copy of the weights already exists in GPU memory. Rather than treating every replica as an independent cold start, MX selects the fastest available source and transfer path.
Why it matters
Checkpoint sizes now commonly reach hundreds of gigabytes or even a terabyte, and weight movement across the cluster is frequent: cold starts pulling weights from remote storage into GPU memory, autoscaling and rolling updates populating new replicas, and reinforcement learning (RL) post-training moving updated weights from trainers to rollout workers. Time spent moving weights delays useful work; MX aims to reduce that recurring overhead.
How ModelExpress works
MX follows two core principles: (1) if a serving peer already holds compatible weights in GPU, copy them directly GPU-to-GPU over P2P RDMA using the NVIDIA Inference Xfer Library (NIXL), avoiding redundant access to object storage, local disk, and host memory; (2) when no peer is available, bootstrap from the fastest supported path, streaming from object storage into GPU memory without landing on disk.
Example: DeepSeek‑V4 Pro
MX transfers DeepSeek‑V4 Pro weights and JIT kernel cache artifacts from a serving replica into a fresh replica in under 10 seconds, reducing total startup time from 8 minutes to 1 minute 44 seconds.
Optimizing load paths
Every new worker gets weights from one of three sources: remote object storage (e.g., Hugging Face or S3), local storage, or another worker already serving the model. MX bootstraps the first worker from the fastest storage-backed path (streaming or local fast storage) and then automatically prefers GPU-to-GPU P2P RDMA for subsequent compatible workers, falling back to storage only when no peer exists.
From remote object storage to GPU: Model Streamer
If a checkpoint sits in a cloud bucket and you prefer not to provision a disk cache tier, MX’s Model Streamer pulls safetensors through a reusable CPU staging buffer directly into GPU memory so the checkpoint never lands on local disk. Model Streamer fetches tensor ranges concurrently across checkpoint shards with a multithreaded tensor reader and pipelines remote reads with GPU placement: completed tensors are passed to the inference engine while later tensors are still being fetched. This overlaps storage, network, and GPU copy paths while bounding host memory usage.
In tensor-parallel deployments ranks divide remote reads (typically via NCCL) and share results instead of each rank downloading the entire checkpoint. MX connects this distributed stream directly to the inference engine’s weight loader so the first worker is prepared to become the P2P source for later replicas.
Cluster ingress: download once, not N times
When a cluster uses a shared disk cache tier (e.g., Kubernetes persistent volumes), MX’s Model Cache Service ensures the fleet populates it only once. For example, if 10 replicas concurrently request the 806 GiB DeepSeek‑V4 Pro model, naively they would pull about 8 TiB of identical data over the network; MX collapses those requests into a single coordinated download using an atomic claim in the Metadata Store so all replicas reuse the cached copy.
Local storage to GPU: GPUDirect Storage and ModelStreamer
If the system supports GPUDirect Storage (GDS), MX uses NIXL’s multithreaded GDS backend to read checkpoint files directly from local storage into GPU memory, bypassing host memory staging. If GDS is unavailable, MX’s ModelStreamer can load local checkpoints by reading safetensors concurrently into a configurable CPU buffer while completed tensors are moved to GPU, overlapping disk I/O with GPU placement and benefiting from the OS page cache.
Fetching from a serving peer: the key feature
Once a replica is already serving the same model, its weights are resident, post-processed, and laid out in GPU memory for the inference engine. MX treats that replica as a live weight source: after confirming compatibility, it transfers tensors directly from the source GPU to the target GPU. The new replica then joins the source pool, enabling subsequent replicas to scale out via GPU-to-GPU fan-out instead of repeated cold loads.
The MX control plane discovers compatible peers, exchanges transfer metadata, and tracks source readiness (using Redis, Kubernetes CRDs, or k8s-service metadata backends), but it never handles the weight bytes themselves. On the data plane MX uses NIXL as a default transfer engine with pluggable backends for networks such as Infiniband, RoCE, NVLink, EFA, etc. MX’s transport interface allows libraries like fabric-lib and standalone Mooncake to integrate with MX.
Before any transfer, MX computes an mx_source_id from the model and runtime settings that determine tensor layout, and it considers only peers with a matching ID.
Reducing NIXL memory registration overhead
RDMA requires registering the GPU memory backing a tensor (ibv_reg_mr, returning an rkey). Large models may have tens of thousands of tensors; registering each tensor separately is slow. Two opt-in strategies reduce registration cost:
- Pool registration: register each underlying cudaMalloc allocation once instead of each tensor, cutting registration counts by roughly 80–99% on typical models.
- VMM arena registration: install a CUDAPluggableAllocator that routes load-time allocations into a single 16 TiB virtual-address arena, then register the whole used range once as a dmabuf-backed region; registration collapses from one call per tensor to a single call.
The team measured average NIXL registration time using DeepSeek‑V4‑Pro with TP=8 on the vLLM engine to compare approaches.
Runtime path selection and safe fallback
At startup MX probes environment capabilities and skips unsupported paths. The first applicable strategy runs in priority order: P2P RDMA -> ModelStreamer -> GDS -> default loader (host-staged POSIX I/O). If a path is unavailable or fails before model state is modified, MX falls through automatically; if failure occurs after weights start landing, it reinitializes the model to avoid serving partially written weights. P2P retries only alternate peers for metadata failures before transfer begins; the native loader is the final fallback.
This capability-driven design keeps MX hardware- and platform-agnostic while enabling platform-specific fast paths where supported.
End-to-end results (test configuration)
The team ran DeepSeek‑V4‑Pro on an 8×B200 GPU node with NVIDIA ConnectX‑7 NICs and compared total model loading time across cold-start scenarios. Each replica used vLLM 0.23.0 with TP=8 and --enable-flashinfer-autotune. MX’s P2P RDMA and ModelStreamer paths substantially reduced total load time compared to HF/S3 streaming or disk-based loads.
Beyond weights: inheriting compiled kernels
Loading weights into GPU memory is necessary but not sufficient: the inference engine JIT-compiles and autotunes kernels (torch.compile, Triton, DeepGEMM, TileLang, etc.) and captures CUDA graphs for the exact model, dtype, quantization, and GPU. For models like DeepSeek‑V4 Pro this compilation can take several minutes and become the dominant startup cost once MX shortens weight-loading latency.
MX’s Artifact Transfer API packages file-backed artifacts, transfers them directly between registered host-memory buffers over NIXL’s CPU-to-CPU RDMA path, then verifies and installs them to the target engine’s cache directory. This removes the need for a shared ReadWriteMany (RWX) volume in Kubernetes, and an artifact-specific mx_source_id prevents reuse across incompatible replicas. In tests transferring Triton/DeepGEMM/TileLang/CuTe DSL/FlashInfer caches, artifact transfer significantly reduced total startup time.
RL post-training: receiver-driven refit
RL post-training requires updated weights to flow from trainers to rollout workers between steps. MX drives receiver-driven refit in four stages: Publish (trainer advertises shards it owns), Discover (rollout worker looks up sources for a weight version), Plan (map published ownership to the receiver’s target layout), Pull/convert/load (the receiver issues one-sided reads against those sources). MX provides the building blocks for this flow and customers are evaluating integrations; the team is also testing delta weight diff refits for cross-cluster transfer.
Integrations and roadmap
MX has native integrations with vLLM and SGLang and supports serving frameworks including Dynamo and llm-d. The Dynamo open source community continues work toward deeper TensorRT-LLM integration and broader inference capabilities.
Acknowledgments
ModelExpress is a team effort. The announcement thanks the MX team and specifically Zhongdongming Dai and Tanushriya Singh for core work, plus Itay Neeman, Anish Maddipoti, Istvan Haller, and Omri Kahalon for technical guidance, and Will Eaton at Red Hat for support on the llm-d integration.
