NVIDIA Mission Control 2.5 extends the autonomous recovery engine to Kubernetes-based AI workloads managed with NVIDIA Run:ai. Previously, this automatic recovery capability was limited to Slurm-based high-performance computing (HPC) workloads. The update is intended to help AI factories scale resiliency and uptime by making recovery processes more autonomous and scheduler-aware.
Why this matters now
Training large AI models often runs distributed jobs across hundreds or thousands of GPUs for days or weeks. At that scale, hardware anomalies, software errors, network timeouts, and resource contention must be handled proactively to avoid lost compute and missed business opportunity. Traditional manual intervention and reactive monitoring do not scale when many concurrent multi-node workloads share infrastructure.
Core elements of the autonomous recovery engine
The Mission Control autonomous recovery engine aims to make AI factories self-healing: failures are detected quickly, recovery decisions are made based on workload context and failure type, and jobs resume automatically with minimal disruption. The objective is to identify what failed, isolate anomalies for maintenance, and return workloads to productive compute with as little wasted time as possible.
The engine consists of two services:
- autonomous job recovery
- autonomous hardware recovery
These services form part of a three-layer control loop together with the workload scheduler (Slurm or Kubernetes). Each layer has a distinct role but they coordinate to enable faster, context-aware recovery.
The three roles in the control loop
-
Kubernetes scheduling layer: NVIDIA Run:ai is the default Kubernetes scheduler included in Mission Control. It manages Kubernetes-based AI workloads, queues, policies, and places jobs onto available capacity.
-
Autonomous job recovery: this layer provides workload context. It watches running jobs, tracks which node or GPU is executing each job, and uses failure history and job topology to classify the appropriate recovery path (restart, retry, re-queue, or re-placement). Integration with the resource manager determines how that guidance is enacted.
-
Autonomous hardware recovery: this layer monitors infrastructure health. It observes nodes and GPUs, removes anomalous hardware from scheduling eligibility when necessary, and runs diagnostics or remediation so hardware problems do not spread into new jobs.
Separating these responsibilities makes the approach portable: job recovery does not need to repair hardware, hardware recovery does not need to know every training framework, and the scheduler can route workloads to ready capacity without waiting for root-cause analysis.
Failure paths in practice
The control loop responds differently depending on the failure signal detected by the autonomous recovery engine:
- Hardware fault during training
If a GPU begins failing during a distributed training run—manifesting as ECC errors, a hung worker, or a communication timeout—recovery splits into parallel tracks. Autonomous job recovery captures job context and identifies the suspect node or GPU; the Kubernetes scheduler then restarts or reschedules the job on healthy nodes. Simultaneously, autonomous hardware recovery isolates the problematic node, records the reason, and launches diagnostics. This reduces downtime because the workload does not wait for a full root-cause analysis before recovery begins, and the faulty node is prevented from being handed to the next job.
- Transient software fault
Not every failure warrants hardware remediation. If a worker exits, a recoverable timeout occurs, or coordination is briefly lost while the underlying node is healthy, autonomous job recovery will avoid overreaction: the system retries according to scheduler policy instead of quarantining hardware that does not need repair. Autonomous hardware recovery continues background monitoring but only intervenes if new evidence points to hardware issues. This restraint prevents unnecessary operator alerts and preserves scheduling pool capacity.
- Early detection of hardware degradation
Autonomous hardware recovery can detect infrastructure degradation before it causes job failures—repeated GPU health warnings or connectivity problems, for example. In that case the service can proactively remove the node from scheduler eligibility and start diagnostics before any training job lands there. The job recovery service may never need to act because the suspect capacity is removed from circulation ahead of disruption. This prevention reduces the likelihood of future outages.
Operational impact
Extending Mission Control’s autonomous recovery engine to NVIDIA Run:ai–managed Kubernetes workloads is a step toward self-healing AI factories. The model shifts resiliency from scheduler-specific incident playbooks to a scheduler-aware operating model.
- For platform teams, Mission Control 2.5 provides a consistent recovery model across both Kubernetes and Slurm estates: faulty nodes are isolated, annotated, diagnosed, and returned to service only when validated.
- For AI practitioners, both Slurm and Kubernetes users benefit from simpler recovery: jobs can restart, retry, or requeue according to their scheduler’s policy while the autonomous engine supplies context to avoid repeated failures on infrastructure that needs service.
- For the business, improved utilization of accelerator capacity follows: less time is spent stalled in ambiguity and more time is spent keeping training workloads productive.
Release and additional highlights
Mission Control 2.5 is available today. Existing NVIDIA Mission Control users can consult the documentation hub for upgrade guidance. In addition to extending autonomous recovery to Kubernetes-based workloads, other release highlights include:
- New AI-enabled network anomaly detection service (beta)
- Improved authentication user experience across services using central Keycloak
- Additional security and Kubernetes hardening updates
- Early access for NVIDIA Vera Rubin NVL72
The release positions Mission Control as an enterprise-ready solution for workload orchestration, end-to-end monitoring, and autonomous recovery across heterogeneous AI infrastructure.
