NVIDIA DSX OS is the open-source software layer of the NVIDIA DSX AI Factory Platform that provides tools for configuring, certifying and operating GPU-accelerated Kubernetes infrastructure. The layer includes three main open-source projects: NVIDIA AI Cluster Runtime (AICR), NVIDIA Cluster Readiness Engine (NVCRE), and NVSentinel. All three projects are released under the Apache 2.0 license and accept external contributions.
Why this matters for operators
A single underperforming GPU can slow a distributed training job without triggering simple health checks. Cluster state changes over time as nodes are replaced, drivers are patched, and capacity is added; without repeated validation, a previous certification can become stale. That visibility gap wastes compute on failing nodes and holds healthy nodes idle out of caution. DSX OS aims to close that gap by providing reproducible configuration, workload-based certification, and continuous operational monitoring with quarantine and revalidation.
Configure: AICR — a reproducible baseline
NVIDIA AI Cluster Runtime (AICR) captures the cluster software baseline in version-controlled recipe files, pins component versions, encodes dependency ordering, and renders artifacts that existing deployment tooling applies. AICR helps detect configuration drift, keeps node configurations consistent after repairs, and improves failure reproducibility.
Example recipe creation command:
aicr recipe --service eks --accelerator h100 --os ubuntu --intent inference --platform dynamo -o recipe.yaml
Each component entry in the recipe pins a version, declares dependencyRefs, and includes healthCheckAsserts blocks that run detailed checks after installation. Rendering produces bundles for deployers such as helm, Argo CD, Flux, or helmfile:
aicr bundle --recipe recipe.yaml --deployer argocd --output ./bundles
Because the recipe is a file, teams can use version history, code review, and CI checks. Commands such as aicr validate and aicr diff --fail-on-drift enable pipeline gates that fail on drift rather than only reporting it.
Certify: NVCRE — workload-based readiness validation
NVIDIA Cluster Readiness Engine (NVCRE) validates whether a cluster can run production workloads by executing AI training jobs, NCCL communication tests, and diagnostic suites. This workload validation targets goodput (successful work done) rather than basic service liveness.
A Certification manifest specifies the target nodes, test categories and performance thresholds. Example snippet:
apiVersion: nvcre.nvidia.com/v1alpha1 kind: Certification spec: target: nodeSelector: nvidia.com/gpu.present: "true" categories: - domain: communication variant: nccl-all-reduce - domain: training variant: nemotron5-8b options: thresholds: busBandwidthGBps: "value >= 300"
Run and report commands:
nvcrectl certification run --cert-file certification.yaml --wait nvcrectl certification report gpu-cluster-cert -n <namespace>
The report shows each category’s status, runtime, measured bandwidth and the reasons for any failed nodes. Thresholds turn a run into a performance gate — without thresholds, a node performing below target might still pass. NVCRE’s catalog includes NCCL variants, NVIDIA DCGM level‑4 diagnostics, and pretraining workloads using NVIDIA NeMo with large models. You can also wrap your own containerized tests as WorkloadRun resources.
NVCRE can bisect failing groups: if an all-reduce across many nodes shows poor bandwidth, the engine splits the group and reruns tests until it isolates suspect nodes and reports failure reasons. Best practice is to run NVCRE before handing capacity to tenants and after repaired hardware rejoins the pool.
Operate: NVSentinel — detection, quarantine and revalidation
Once a cluster is certified and running workloads, NVSentinel provides continuous monitoring and lifecycle controls. It observes existing signals (DCGM health checks, Xid errors, NIC states, and cloud maintenance events) without consuming GPU time. By default NVSentinel installs in monitoring mode and only reports node conditions; quarantine, draining and remediation can be enabled later.
Fault Quarantine rules use Common Expression Language (CEL) to match health events and nodes. Example ruleset fragment:
fault-quarantine: ruleSets: - enabled: true name: "GPU fatal error ruleset" match: all: - kind: "HealthEvent" expression: "event.agent == 'gpu-health-monitor' && event.componentClass == 'GPU' && event.isFatal == true" cordon: shouldCordon: true taint: key: "nvidia.com/gpu-error" value: "fatal" effect: "NoSchedule"
A cordon or taint provides a scheduler-visible signal that can integrate with existing alerting and lifecycle tooling. NVSentinel includes circuit-breaker limits to cap how many GPU nodes can be quarantined in a sliding window. The health events analyzer looks at failure history so that nodes failing repeatedly are escalated based on accumulated patterns rather than single events. Some health events recommend CONTACT_SUPPORT for manual investigation.
Continuous readiness: the validation lifecycle loop
NVSentinel’s lifecycle controller can automatically create ValidationRequest resources when a node joins the cluster or returns after remediation. It gates scheduling with a cordon or taint, provokes NVCRE tests, and only removes the gate once validation passes. This prevents a repaired node that still fails tests from rejoining the pool silently.
For in-place driver upgrades, operator upgrades or fabric changes, revalidation is not automatic; teams should trigger certification runs from their pipelines (for example by failing on drift with aicr diff --fail-on-drift). A sample ValidationRequest that targets specific nodes and tests looks like:
apiVersion: nvsentinel.nvidia.com/v1alpha1 kind: ValidationRequest spec: nodes: - name: node-a - name: node-b tests: - nccl-all-reduce - nemotron-5b
Status can be queried with kubectl get validationrequests and node labels filter for validation-failed state. A later successful ValidationRequest clears prior failures and automatically releases gates.
NVCRE is responsible for attribution (which nodes failed and why); NVSentinel owns quarantine and remediation. Keeping that separation avoids double-cordoning and allows platform teams to consume certification results and implement their own lifecycle policies without forking the projects.
Operational impact and responsibilities
Together the tools make configuration, validation and recovery repeatable. NVSentinel can coordinate quarantine, repair and revalidation, reducing manual handoffs. Operators still retain responsibilities such as approving baseline changes, selecting appropriate tests and thresholds, investigating failed results, and defining quarantine, remediation limits and escalation procedures.
A high-level mapping of responsibilities:
- Maintain baseline: AICR provides version-locked recipes; operators approve and deploy changes and run drift checks.
- Validate capacity: NVCRE runs workloads and diagnostics; operators pick tests and act on failures.
- Recover nodes: NVSentinel can coordinate quarantine and remediation; operators define rules and handle unresolved faults.
- Validate infra changes: operators should trigger validation after in-place changes.
Getting started
Refer to the project repositories for installation instructions and examples:
- NVIDIA AI Cluster Runtime (AICR)
- NVIDIA Cluster Readiness Engine (NVCRE)
- NVSentinel
NVIDIA also runs workshops and demos (for example at conferences) and maintains community channels such as a DSX AI Factory Slack and the DSX AI Factory GitHub where contributors and users can continue the discussion.
By combining AICR, NVCRE and NVSentinel, DSX OS provides reproducible configuration, workload-based certification, and operational controls that help reduce hidden performance and availability risks in GPU Kubernetes clusters.
