Adapting Kubernetes to new GPU hardware such as NVIDIA Vera Rubin NVL72 is complex: drivers, kernels, runtimes, networking, operators and schedulers must interoperate and be tuned for the hardware and workloads. Small mismatches can prevent jobs from running or limit performance.
The NVIDIA AI Cluster Runtime (AICR), part of the NVIDIA DSX AI Factory Platform, addresses this by capturing tuned configurations as versioned recipes, generating deployable bundles for existing tools, and validating the running cluster. AICR’s new Vera Rubin recipes aim to make Kubernetes clusters ready on Day 1 for Vera Rubin NVL72 compute trays.
What changes compared with GB200 NVL72 — planning upgrades
Vera Rubin NVL72 continues the NVIDIA MGX rack-scale approach but introduces several user-visible differences that platform teams should plan for:
-
GPU driver: Vera Rubin requires a new driver branch. The validation used the host-installed R615.23 open-kernel-module driver. Early support targets R610 and official platform support is expected in R620. AICR’s GB200 baseline was R580.173.02; R615.23 is the current tested version in the Vera Rubin bring-up, not a formal minimum for general availability.
-
DOCA and NIC/DPU drivers: Vera Rubin replaces ConnectX-7 NICs and BlueField-3 DPUs with NVIDIA ConnectX-9 SuperNICs and NVIDIA BlueField-4 DPUs. DOCA 3.3.0 is the first GA release supporting ConnectX-9; the validated system image used DOCA-OFED 3.3.0. In AICR testing, host-installed NIC drivers and DraNet were used for pod allocation; Network Operator compatibility is a separate qualification.
-
Device topology: the NIC/GPU/DMA topology differs from GB200, so GPU-to-NIC affinity and NUMA placement need checking for I/O-heavy workloads (RDMA, GPUDirect Storage). Both reference trays have two CPUs and four GPUs, but the Vera Rubin trays expose eight ConnectX-9 PCI device groups with 32 functions versus four ConnectX-7 adapters on GB200. NUMA affinity remains: GPUs 0–1 -> NUMA 0, GPUs 2–3 -> NUMA 1. GPU–NIC connectivity is NODE (within a socket) or SYS (across sockets).
-
IMEX: multi-node NVLink workloads still use IMEX. For DRA-managed IMEX, disable and mask the host static nvidia-imex service to avoid ownership conflicts. DRA v0.5.0 added an alpha host-managed option with domain-only isolation; the validated clusters used driverManaged mode.
-
NVIDIA MIG profiles: Rubin introduces new MIG profiles that R615.23 advertises. AICR Vera Rubin recipes use whole-GPU allocation and do not configure MIG; MIG was not part of the validation.
These deltas are encoded in the Vera Rubin recipe family so that aicr recipe and aicr validate reflect the validated combination rather than requiring operators to rediscover each change.
Validation on Vera Rubin NVL72
End-to-end validation used the AICR runtime bundle on Vera Rubin clusters with GPU Operator v26.7.0, DRA driver v0.5.0, and a host-installed R615 driver. Both training and inference recipe variants validated for deployment and conformance. For training, a performance gate ran a cross-node NCCL all-reduce over the NVLink fabric through an IMEX ComputeDomain and checked bus-bandwidth against a floor. GPU pods ran cleanly and IMEX channels were injected into claiming pods across multiple nodes.
Early-silicon realities surfaced during bring-up: one early Vera Rubin cluster had a hardware/fabric issue that left one compute tray out of the NVLink fabric; after reseating the hardware, all GPUs joined a single clean fabric clique.
From bring-up to reproducible recipes
Validating components is only part of the work. Standing up a working Vera Rubin cluster requires selecting driver mode, container-runtime wiring, component versions and platform-specific settings. AICR captures those selection decisions as a versioned recipe so the configuration used on real Vera Rubin hardware is explicit, reproducible and verifiable across clusters.
The Vera Rubin recipe family (service: rke2; accelerator: Vera Rubin NVL72 (vr200); OS: Ubuntu) ships as Preview with training, inference and NVIDIA Dynamo inference variants. It encodes bring-up decisions such as:
-
Host-managed driver: the Vera Rubin system image pre-installs the arm64 open-kernel-module driver, so the recipe runs the GPU Operator with driver.enabled: false while the container toolkit remains operator-managed. The recipe will shift to operator-managed driver when official builds arrive.
-
DRA driver with ComputeDomains: IMEX-backed ComputeDomains are enabled for the NVL72 fabric while the device plugin retains whole-GPU allocation.
-
Platform floors and quirks: these are encoded once to avoid repeated discovery. Required items include Kubernetes >= 1.34.1 (resource.k8s.io/v1 DRA API and the RKE2 containerd path needs 1.34.1+rke2r1), Ubuntu 26.04 with the Vera-optimized arm64 64k-page kernel, and small tweaks such as raising the NFD topology updater’s default 60Mi memory limit to avoid OOM kills on multi-terabyte NVL72 workers.
The first bundle was deployed to a platform combination that none of its components had previously been validated on together (arm64, Ubuntu 26.04, 64k-page kernel, RKE2, Vera Rubin) and moved from cluster handoff to a healthy runtime in hours. A second cluster reproduced the result the next day; repeatability is the central benefit of the recipe.
Live workflow example (AICR v1.0.0)
The actual AICR workflow executed on a Vera Rubin NVL72 reference cluster has four steps:
-
Generate the recipe: offline generation resolves criteria in seconds and emits recipe.yaml with the resolved components and overlays.
-
Render the deployment bundle: aicr bundle produces an ordered set of per-component Helm charts and a deploy script (in the example, 61 files and 14 components were produced).
-
Snapshot and validate: aicr snapshot captures the cluster state, and aicr validate checks readiness constraints and runs in-cluster validators across deployment, conformance and performance phases. All phases passed on the test cluster.
-
Cross-node ComputeDomain test: a two-node ComputeDomain was created and IMEX channels were claimed by pods on each compute tray. Note: for host-managed IMEX images the host nvidia-imex service must be disabled and masked because host-managed and DRA-managed IMEX are mutually exclusive today. Also, GPU Operator v26.7.0+ ships a stricter computedomain CRD that requires numNodes explicitly on fresh installs.
The end-to-end sequence (recipe → bundle → snapshot → deployment+conformance) completes in about five minutes on a live cluster; the performance phase adds the NCCL job runtime.
Validation evidence and key metrics
The AICR deployment, conformance and performance validations across NVL72 reference environments, paired with the ComputeDomain test, produced signed validation evidence. The training recipe includes a cross-node NCCL performance gate — the first performance validation on Vera Rubin silicon. The measured NCCL all-reduce result was 744.71 GB/s bus bandwidth at 16 GiB.
The inference variant (Dynamo platform, inference gateway, gang scheduling, DCGM metrics) passed the same deployment and conformance checks on the identical cluster.
Some inference performance work remains: vLLM merged Rubin (sm_107a) support in v0.27.0, but the SM107 kernel path requires CUDA >= 13.4 to be effective. Published vLLM images through v0.28.0 are still built against CUDA 13.0.3, and ai-dynamo v1.4.2 pins vllm-openai v0.26.0 (which predates the Rubin changes). Therefore, a VR200 inference-performance floor needs a published runtime image built with CUDA >= 13.4 and validated on real hardware.
Feedback loops into upstream projects
Running AICR validation on Vera Rubin exposed an unmapped Rubin architecture value in the go-nvlib packages that caused a crash-loop in the DRA kubelet plugin. Because support propagates from go-nvml into go-nvlib and then into the GPU Operator, device plugin, MIG Manager and DRA plugins, the fix (go-nvlib 0.11.0 → 0.12.0) was propagated and shipped in DRA driver v0.5.0. This fix reached a validated build on Vera Rubin hardware in about three days.
Validation also revealed that host-managed IMEX daemons and DRA-managed ComputeDomains cannot share a node today; that finding influenced the host-managed IMEX option added in DRA v0.5.0.
Where to get started
The Vera Rubin recipes are published in the NVIDIA/aicr repository as v1.0.0 and register the rke2 service and vr200 accelerator in the public catalog. Try the Preview recipes with these commands:
-
Generate the training recipe: aicr recipe --service rke2 --accelerator vr200 --os ubuntu --intent training
-
Generate the Dynamo inference recipe: aicr recipe --service rke2 --accelerator vr200 --os ubuntu --intent inference --platform dynamo
You can follow progress, review code or open issues on the related GitHub projects: NVIDIA/go-nvml, NVIDIA/go-nvlib, kubernetes-sigs/dra-driver-nvidia-gpu and NVIDIA/aicr.
AICR’s intent is to make the Vera Rubin hardware transition faster, reproducible and verifiable for platform and cluster teams.
