NVIDIA Recovery is an open-source workload-level control-plane service in the NVIDIA DSX OS stack that automates detection, attribution, isolation and restart of interrupted distributed AI training jobs across a cluster. As distributed training scales, individual hardware or software disruptions — for example a stalled rank causing an NVIDIA Collective Communications Library (NCCL) timeout, a GPU node that stops responding to the scheduler, or a straggler task — can together create significant downtime and wasted GPU-hours. Continuous, automated recovery reduces those losses and shortens time-to-train.
Where it fits in the stack
NVIDIA Recovery provides the cluster-level workload recovery layer and integrates with existing health and framework tooling:
- Node health and infrastructure monitoring: in Kubernetes NVSentinel monitors GPU metrics and XID events and can cordon and drain faulty nodes. Many Slurm sites use custom prolog/epilog scripts; the slurm-operator (Slinky project) injects these when Slurm runs on Kubernetes.
- In-job recovery and hang detection: the NVRx library gives per-job facilities (recovery, hang detection, straggler mitigation). NVIDIA Recovery complements this by managing cross-workload lifecycle actions, restart placement and node exclusion.
- Optional diagnostics: NVIDIA Attribution can provide deeper cross-layer verdicts to enrich the persistent audit trail.
The components are modular: you can adopt parts independently or integrate your existing health stack with limited changes.
Why automation is necessary
Rare per-GPU failures compound at scale. If a single GPU has a 99.9% daily uptime, a job spanning 10,000 GPUs finishes the day without interruption with probability roughly 0.999^10000 ≈ 0.005%. In an 11-month study of 24,000 A100 GPUs at Meta, hardware failures interrupted fewer than 1% of jobs while those interrupted jobs accounted for 19% of GPU runtime — the largest and longest-running jobs are most exposed.
Automated recovery is beneficial for both many small jobs (where interruptions accumulate) and large jobs (where each interruption affects many GPUs). It reduces operational burden and the compute cost of recovery.
Production observations
Between July 9 and Oct. 9, NVIDIA Recovery monitoring on one training cluster covered nearly 200 jobs per day and recorded about 8,550 anomaly detections. Mean detection and recovery-action latencies were 69 and 94 seconds, respectively. These automated, unattended responses reduce reliance on individual engineers; however, productive training resumes when the scheduler and checkpoint restore complete.
The automated recovery loop
NVIDIA Recovery coordinates recovery in four stages; actions depend on the detected interruption and the configured policy:
- Detect: the Anomaly Detector uses application logs, scheduler signals and GPU metrics to identify crashes, hangs and stragglers. It recognizes PyTorch and NCCL failures, CUDA errors and interconnect failure patterns; missing progress signals or communication-hang patterns indicate hangs, while sustained throughput degradation vs. recent baseline indicates stragglers.
- Attribute: scheduling signals and logs gathered during detection classify the interruption and guide recovery. Integrating NVIDIA Attribution yields deeper diagnosis to distinguish hardware faults, system-software issues and application bugs. Operators may also attach their own attribution service.
- Isolate: the Anomaly Remediator applies the recovery recommendation, excluding implicated nodes from restart attempts or stopping retries when unproductive. It tracks implicated nodes across attempts to avoid repeat failures. A watchdog service allows normal scheduling to proceed if a recovery decision takes too long or fails outright.
- Restart or Stop: via the Advisor API the scheduler acts on the recommendation to restart the workload or stop retries. On restart, the application resumes from its last saved checkpoint using its existing restart logic. If the job cannot save a new checkpoint after a configurable number of attempts, further retries can be stopped. The audit trail records restart and stop decisions as distinct events.
Scheduler integration and deployment modes
NVIDIA Recovery runs as a control-plane service and exposes an HTTP or gRPC API for scheduler integration. It currently supports Slurm; Kubernetes support is on the roadmap. Recovery recommendations use a common API contract and a cluster-side controller translates them into workload-manager actions, avoiding changes to application code.
For Slurm the reference Advisor Controller is available as a Helm chart and supports two modes:
- In-cluster mode: the controller runs inside the cluster.
- SSH mode: the control plane connects to Slurm remotely via SSH, without deploying an in-cluster service — useful when adding a new in-cluster component requires a change window or security review.
Each recovery recommendation is recorded before the controller acts. If the controller restarts mid-recovery, it checks the scheduler state before proceeding to avoid repeating actions.
Monitor mode, the Recovery Console and the audit trail
New installations default to monitor mode: NVIDIA Recovery publishes advice and records what it would have done without taking scheduler actions. This allows operators to review recommendations and validate signal-to-noise before enabling autonomous action.
The installation workflow includes:
- Install the Helm chart: NVIDIA Recovery is distributed as a single umbrella Helm chart with a bundled MySQL sub-chart for quickstarts. In production, point the chart at a managed MySQL service. The operator deployment guide documents supported Kubernetes, Helm and Slurm versions.
- Validate signals: point at your Loki or Victoria Logs endpoint and Prometheus scrape target, then run the log ingestion configurator to test signals against the NVIDIA Recovery contract and map label differences.
- Opt a workload in: add a directive to your sbatch submission script, for example:
#SBATCH --comment='{"nvidia-recovery":{"environment":"prod","mode":"observe"}}'
NVIDIA Recovery then monitors that workload; after validating recommendations in the Recovery Console, switch to action mode and set acknowledgeSchedulerActions to true in the Helm values to enable automatic recovery.
Every detection and action is written to a persistent, queryable audit trail: matched detection pattern, action taken (kill, requeue, release, node exclusion), timestamp, workload identifier and NVIDIA Attribution verdict if available. Policy-engine and watchdog actions are recorded together so there are no gaps.
The Recovery Console is the operator dashboard showing live workload state across scheduler types, inline anomaly flags and a per-workload timeline of detection and recovery events.
Source, license and early adopters
The code and issue tracker live in the dsx-ai-factory/NVIDIA-Recovery GitHub repo under the Apache 2.0 license. The quickstart can run without a physical GPU. IBM Research used monitor mode on their Slurm cluster and enabled autonomous actions after validation. Verda, an NVIDIA Cloud Partner, deployed with VictoriaLogs and contributed a VictoriaLogs connector back to the repository.
NVIDIA Attribution is the recommended companion deployment for richer diagnostics but is optional and ships alongside NVIDIA Recovery.
Roadmap and next steps
Current support covers Slurm jobs, including Slurm on Kubernetes via slurm-operator. Upcoming capabilities include Slurm job array recovery (per-task recovery for hyperparameter sweeps that stops when failures look systemic) and native Kubernetes recovery starting with JobSet and Kubeflow PyTorchJob using the Workload Map project. Future work also covers sharper failure classification, clearer records of every recovery decision in the dashboard and API for observability integrations, and progress toward a stable release. Community feedback (issue upvotes and comments) will influence priorities.
Conclusion
NVIDIA Recovery automates the recovery loop that manual processes often leave to on-call engineers, reducing manual effort, shortening detection-to-action times and maintaining an auditable record of recovery decisions. To get started, clone the GitHub repository, run the configurator against your telemetry stack, opt a workload into monitor mode, and observe the loop on the first interruption.
Acknowledgments
Contributors to the project and post include: Abhishek Jha, Alex Kondratenko, Amy You, Andrew McHarg, Ashwin Poojary, Ashwini Awatiger, Bob Schatz, Collin Neale, Dave Yao, Deepak Narayanan, Dilpreet Singh, Divyansh Jain, Dong H. Ahn, Dorrin Poorkay, Evan Briones, Gargi Prasad, Haim Elisha, Hamzeh Doostie, Harmandeep Singh, Harri Lin, Himanshu Soni, Ian Gibbs, Jason Sewall, Kumar Anik, Maer Rodrigues de Melo, Megan Chao, Mike Houston, Orlando Gentil, Oscar Bejarano Ardila, Pavan Kumar Gondhi, Prateek Gera, Rajan Singh, Rakshit Sanadhya, Rajasi Purandare, Ripple Patel, Rick Izzo, Ruoxi Zhang, Satish Pasumarthi, Shelby Thomas, Stefania Alborghetti, Tony Wang, Vincent Pun, and Yugal Jain.
