AI factories—large AI data centers running many multi‑node training jobs—combine some of the most complex technical layers in production: hundreds of thousands of GPUs, NVIDIA Spectrum‑X Ethernet and NVLink fabrics, SuperNICs, DPUs, distributed schedulers and multi‑week training runs. These layers act as a single production system, and a single degraded GPU, interface, storage device or switch path can reduce throughput across many otherwise healthy resources.
The failures that damage yield are often subtle: a GPU a few degrees hotter than peers, an NVMe that loses performance only under sustained load, or a switch that produces pause‑frame storms. Individually these signals may not cross static thresholds, but together they create stragglers, longer step times and delayed checkpoints.
How Minerva improves operations
NVIDIA Minerva is an AI‑native AIOps platform that collects telemetry across the full stack, maps infrastructure to active jobs in a dynamic knowledge graph, applies purpose‑built machine learning, and correlates dependent symptoms into high‑fidelity incidents. A built‑in AI agent called Sherlock and an MCP server that connects to external agentic tools let operators query the same live evidence conversationally. The practical goal is to preserve effective GPU hours and keep model training progressing.
Why conventional monitoring fails at AI factory scale
Rule‑based monitoring assumes a known metric crossing a known boundary indicates a known condition. That model handles clear failures but struggles with “gray failures,” where components remain available and the issue appears only under load or relative to peers. At large scale, one physical fault can produce many downstream metrics and alerts across devices, creating alert amplification that forces manual correlation by on‑call engineers.
For operations, the unit that matters is the workload: teams must know which job is affected, which resources it occupies, where those resources are in the physical and logical topology, and how current behavior differs from a healthy baseline. That context must be kept current as jobs start, stop and nodes are drained or returned to service.
Building a live model of infrastructure and jobs
Minerva ingests metrics, logs and topology from GPUs, Spectrum‑X and NVLink switches, SuperNICs, DPUs, storage, NVIDIA NCCL, schedulers and AI workloads. Auto‑discovery supplies the relationships raw telemetry lacks, and the resulting knowledge graph maps infrastructure, scheduler and workload elements and updates those relationships as the environment changes. It can span multiple data centers and support navigation from a factory‑wide view to an individual interface.
The topology graph changes investigation paths: from an anomalous interface an operator can find the node, the fabric path it serves, current job placement and nearby components in the same failure domain. From a job ID an ML engineer can trace the workload through GPU, host, NIC, switch and log evidence. Network and compute teams therefore work from the same current causal map instead of reconciling separate inventories.
Detecting gray failures with purpose‑built ML
Minerva processes billions of metric and log data points per hour with supervised and unsupervised models developed for AI‑factory infrastructure and workloads. Supervised detection finds known critical error patterns and can associate recognized conditions with remediation guidance. Unsupervised models learn normal patterns for a hardware or logical component and compare them with peers.
Peer comparison matters because a measurement that looks acceptable in isolation may be anomalous relative to siblings. Multivariate models combine temperature, power, clocks, utilization and workload state to detect degradation before outright failure. NLP‑based log anomaly detection surfaces previously unseen patterns, while supervised matching picks out known critical errors. Detections can feed scheduler workflows to drain affected nodes at safe points.
Collapsing event storms into actionable incidents
Topology‑aware correlation uses the knowledge graph to determine which events share a dependency path and groups downstream symptoms into one incident rather than competing alerts. The result is less alert noise and more context: Minerva can collapse redundant events and related metric series into a single signal with affected topology and workloads attached. NVIDIA testing shows significant reductions in notified events and tickets for supported scenarios. Once Minerva creates a high‑confidence incident, teams can consume it through the platform UI or existing workflows.
Agentic debugging: Sherlock and MCP
Correlated evidence still needs interpretation. Sherlock, Minerva’s native AI agent, is grounded in live telemetry and the knowledge graph so operators can ask in natural language why a specific job slowed during an interval, which components changed state, or whether other nodes share the same failure domain. Sherlock presents supporting evidence and recommended remediation, enabling less experienced operators to follow guided root‑cause analysis.
Minerva’s MCP server lets external agentic tools query the same operational context so teams can incorporate live AI‑factory evidence into their own workflows. Mitigations may remain human‑approved or be automated according to local policy and risk.
Use cases from NVIDIA AI factories
- Detecting a single link degradation in a 400,000‑interface AI factory: An SRE observed one job’s step time gradually worsening without alerts. Minerva’s full‑stack anomaly detection identified the single NVLink connection, linked the switch to the affected training jobs, and enabled the team to drain and reschedule the nodes, isolate the link for repair and restore job throughput in minutes instead of hours.
- SuperNIC proactive maintenance: A node logged a rare PCIe‑level firmware hardware error (APEI, error source 5). Minerva’s rare log anomaly detection flagged the message, posted an alert with remediation steps and the on‑call engineer removed the node from the pool before workloads landed on it—saving hundreds of GPU hours.
- Straggler job due to NVMe power fault: Multiple training jobs were slowed by one host whose NVMe device failed to wake from low‑power state, triggering I/O write errors. Minerva traced the issue to that host; draining the node and replacing the hardware is estimated to have saved 5,530 hours of job runtime.
- Pause‑frame storm: A switch page fault caused a restart that set off congestion, RX pause frames and cascaded link downs, producing spikes in GPU temperature and utilization and generating up to 81 noisy events. Minerva’s topology‑aware correlation linked those 81 events to the affected job and presented them as a single incident, cutting repair time and saving about 54 hours of total cluster uptime.
Operational impact and DSX OS convergence
Internal NVIDIA testing shows Minerva increases mean time between interrupts (MTBI) through proactive maintenance, reduces mean time to repair (MTTR) via automated root‑cause analysis and significantly lowers alert noise—translating into more effective GPU hours. Minerva is being developed in convergence with NVIDIA DSX OS: selected Minerva components beginning with the data collector will become open‑source parts of DSX OS, reusing DSX OS health and resilience components such as NVIDIA Attribution and providing a modular telemetry foundation.
Conclusion
As AI factories scale, the gap between traditional monitoring and operational reality widens. NVIDIA Minerva bridges that gap by unifying telemetry across GPUs, networking, storage and workloads, detecting subtle degradations that static thresholds miss, collapsing noisy event storms into single actionable incidents, and enabling agentic, conversational debugging to shorten triage from hours to minutes.
