NVIDIA has published the Cluster Readiness Engine (NVCRE), an open-source Kubernetes controller that verifies GPU cluster readiness by running real multi-node AI workloads. The tool executes topology-aware tests across node groups, measures results, and reports precisely which nodes caused failures or missed thresholds before production workloads are scheduled.
Why this matters
Standard health checks and telemetry can report all components as healthy while a large synchronous training job still underperforms or fails. A single slow GPU, a network link that degrades under load, or a configuration that routes traffic over a slower path can slow a 512-GPU training run to the speed of its worst rank. Operators often discover such problems hours into a run or when customers file tickets, which can lead to days of manual bisecting while capacity remains idle.
NVCRE aims to make readiness a proven property: it actively generates load with representative workloads and attributes failures to specific nodes and categories so teams find root causes before production jobs land.
How NVCRE works
The project exposes its functionality via an API built on Kubernetes Custom Resource Definitions (CRDs), allowing inspection with kubectl and integration into GitOps workflows.
Layered API and resources
The API defines three hierarchical resources:
- Certification: the top-level resource you create; it names the nodes to test and the categories to run.
- Workflow: manages a single category, applying catalog, platform and GPU overrides, iteration counts, orchestration target, and creating the child Job.
- Job: runs the workload on the target node group, monitors node health, and records measurements and failures.
A certification creates one workflow per category; each workflow creates a job. Job results propagate upward so the workflow reports the test result and the certification groups results by category. This structure attributes each failure to a specific node and category—for example, that gpu-01 had a hardware fault during NCCL while gpu-02 missed a bandwidth target.
Pass/fail criteria and metrics
Pass and fail criteria are expressed in Common Expression Language (CEL) and evaluated against measured metrics. NVCRE ships with no default thresholds; operators define criteria appropriate to their hardware. Illustrative examples for NVIDIA GB200 NVL72-class systems include:
- communication / nccl-all-reduce: busBandwidthGBps: "value >= 900"
- training / nemotron5-8b: goodputRatio: "value >= 0.9", avgTFLOPsPerGPU: "value >= 800"
If a measured metric misses its target, NVCRE sets a ValidationFailed condition separately from whether the workload completed. A job that finishes but misses thresholds is still reported as a failure.
Topology-aware scaling choices
Some failures appear only at particular scales, so NVCRE allows explicit grouping via the testScale field:
- intra-node: test each node independently
- intra-rack: partition by topology domain using labels such as nvidia.com/gpu.clique
- full-scale: include every node in a single group
- diagnose: run adaptive fault isolation
An NCCL test inside an NVLink domain measures NVLink bandwidth; the same test across racks measures the scale-out fabric—different scopes reveal different failure modes.
Adaptive fault isolation
When a failing group’s overall result cannot be attributed to a single node, manual isolation can be time-consuming. With testScale: diagnose, NVCRE performs hierarchical group splitting: it splits a failing group, reruns tests on halves, and continues until minGroupSize is reached. Remaining failing groups are flagged as suspects. A maxConcurrent limit prevents the validation itself from saturating the fabric being measured. The output thus names a small set of suspect nodes and the reason each failed instead of implicating the entire group.
Running arbitrary workloads: the WorkloadRun API
Launching multi-node GPU workloads on Kubernetes requires platform detection, framework-specific runtime configuration, GPU and RDMA resource requests, and cleanup after failures—steps that are repetitive and error-prone. WorkloadRun handles setup: provide a container image, select a framework, and specify the number of nodes.
Example WorkloadRun for an MPI all-reduce test:
apiVersion: nvcre.nvidia.com/v1alpha1 kind: WorkloadRun metadata: name: nccl-all-reduce spec: image: nvcr.io/nvidia/pytorch:26.01-py3 framework: mpi: binary: /usr/local/bin/all_reduce_perf_mpi args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"] mpirunPath: /usr/local/mpi/bin/mpirun numNodes: 4 bandwidthMeasurement: logProfileRef: nccl-bandwidth testType: all_reduce
The framework field supports exactly one of torch (distributed training via torchrun), mpi (NCCL tests and MPI workloads), or exec (an arbitrary command). NVCRE generates the matching Kubeflow TrainingRuntime, injects a shared-memory volume, sets NCCL and platform environment variables, and enables NVIDIA NVLink scale-up networking when supported.
To avoid gang-placement deadlocks on busy clusters, you can opt-in to a gang-aware scheduler such as KAI Scheduler via spec.gangScheduler so pods are held until the whole gang can be placed.
Because WorkloadRun is a plain CRD, external tools can invoke it without adopting the full NVCRE stack: NVCRE provides execution and the caller supplies the test.
Integrating with configuration and monitoring layers
NVIDIA positions NVCRE within a three-layer approach:
- NVIDIA AI Cluster Runtime (AICR) captures and maintains validated, version-locked configuration recipes (drivers, operators, kernels, system settings) so teams reproduce optimized configurations and detect drift.
- NVCRE provides active, workload-driven validation that generates load to reveal failures with little or no telemetry footprint.
- NVIDIA NVSentinel monitors cluster telemetry continuously (DCGM metrics, Xid errors, logs, cloud maintenance events) and can drive quarantine, drain, and remediation workflows.
NVCRE records failed nodes and reasons but intentionally does not cordon, taint, or patch node conditions to avoid conflicting actions. Instead, NVSentinel’s Certification Monitor can translate failed certification results into health events; NVSentinel policies can then quarantine or drain nodes, and a subsequent successful certification can clear the failure signal.
Requirements, quick start and example outputs
Requirements: Kubernetes 1.29+, kubectl, Helm 3.x, and NVIDIA GPU Operator on the target cluster. Some catalog entries (NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72) also require the NVIDIA DRA Driver and ComputeDomain resources. The DCGM level-4 category requires a standalone DCGM service. A gang-aware scheduler such as KAI Scheduler is optional but recommended for busy shared clusters.
Install the CLI and initialize the controller:
$ curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash $ nvcrectl setup init
Run an end-to-end validation and fetch the report:
$ nvcrectl certification run --cert-file certification.yaml --wait $ nvcrectl certification report gpu-cluster-cert -n <namespace>
A sample certification report lists each category’s status, runtime, measured bandwidth and any failed nodes with reasons. Example summary fields include platform, GPU architecture, node count, per-category status (e.g., communication/nccl-all-reduce Succeeded, runtime 3m 56s, full-scale, Nodes/Job 16) and bandwidth tables.
Open source contribution and roadmap
NVCRE is available under the Apache 2.0 license and developed publicly on GitHub. Contributors can report bugs, request features, submit pull requests, or add catalog entries, workload adapters, tests and documentation. Roadmap items mentioned include support for new NVIDIA architectures, inference workloads, and automated lifecycle validation.
The project is part of NVIDIA DSX OS, the operating layer of the NVIDIA DSX AI Factory Platform. Using NVCRE allows teams to validate GPU clusters with shared workflows and a test catalog before production workloads arrive.
