In distributed AI workloads a single fault can generate a cascade of symptoms: a GPU or fabric event may appear as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The exception reported by the workload shows where execution stopped, not where the failure began.
NVIDIA Attribution is an open-source, workload-centric failure-attribution system that correlates workload-scoped logs, metrics, telemetry and scheduler state to produce a structured diagnosis: outcome, root cause (sub_domain), affected nodes, supporting symptoms, and an evidence trail. Being workload-centric means failures are analyzed in the context of the workload rather than as isolated node events. Deterministic rules make results reproducible and customizable; structured output enables integration with existing health and recovery systems.
Production usage and scale
NVIDIA Attribution has been used in production across NVIDIA AI infrastructure for years, analyzing millions of workloads per week across hundreds of thousands of GPUs. In one large RDMA-backed cluster it correlated port errors, connection resets and scheduler events to identify patterns around particular host and switch ports, helping teams target fabric repairs and congestion tuning.
Components and pipeline
The project includes two deployable components—the NVIDIA Attribution Service and the NVIDIA Attribution Blueprint—both driven by a core attribution engine. The Service exposes the engine via an HTTP REST API; the Blueprint connects cluster schedulers and telemetry sources to the Service and writes the structured result.
The Service analyzes one workload attempt at a time. The Blueprint gathers workload context and evidence, the Service extracts and correlates events, and the Blueprint returns a structured verdict. The path has four stages:
- Scope the workload: the Blueprint identifies the workload attempt from scheduler identifiers or from an explicit execution window and node allocation.
- Collect observations: log and metric connectors produce workload-scoped Observation records from application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history, and node and network metrics.
- Extract and correlate events: the attribution engine identifies significant events (GPU XIDs, NCCL failures, scheduler transitions, metric conditions). Deterministic rules evaluate timing and interaction, separate downstream symptoms from the event that caused failure, and find the root cause.
- Return a structured result: the Blueprint combines the verdict with the workload outcome (terminal_state), attributed domain (sub_domain), symptoms, affected nodes, and a record of collector coverage and errors.
From applogs to attribution: an example
A worked example shows how evidence is extracted and interpreted. Consider a multi-rank training workload where one GPU experiences a hard fault (e.g. XID 149) and NCCL errors appear across other ranks. NVIDIA Attribution correlates events across nodes, identifies the XID as the root cause, retains NCCL failures in the symptom chain, and names the responsible node. The workload is reported as a hardware failure rather than a set of independent communication errors.
A failed collective can leave error reports on several nodes; these reports indicate where the workload observed the problem. To explain the failure, Attribution combines them with evidence from other sources and evaluates the symptoms together. In the synthetic example, application logs include three different NCCL errors while system logs on node-02 contain XID 149 with error words decoding to an NVLink signal integrity issue. That evidence proceeds through the rules pipeline.
Extraction and filtering
The Extraction phase runs first, turning raw input logs into typed events with source provenance to be processed by later stages. Filter rules read and write the event stream. In the EARLY stage the HandleXidBurstEvents rule reads the XID burst, decodes the error words, and emits an attribution event for a signal integrity issue on node-02; the three NCCL events remain unchanged.
Refinement to a single attribution
The framework materializes events into per-node symptoms, runs per-node scrubbing, and aggregates surviving symptoms into a job-wide set. In the example all four candidates reach FINAL refinement. The Xid149AndNvlinkFiltering rule retains the signal integrity symptom and demotes other node symptoms; Gb200PriorityRootCause reduces the active job-wide symptom set; outcome mapping produces the hardware-failure attribution.
The final attribution surfaces node-02 and the signal integrity issue while preserving NCCL reports as “interesting” evidence for inspection. When multiple causes fit the evidence, sub_domain can be ambiguous—the diagnosis records uncertainty rather than forcing a definitive label when telemetry is incomplete.
Service and Blueprint architecture
The attribution engine contains the rules engine and core logic. The Attribution Service is a self-hosted HTTP wrapper around that engine which receives normalized observations and provides one verdict per workload. The Attribution Blueprint is a Python reference orchestrator that discovers workloads, resolves ExecutionContext, runs collectors, calls the Service, enriches the verdict, writes the AttributionBlueprintResult, and supplies a read-only dashboard. This split lets teams use the same attribution logic across clusters while adapting connectors to local schedulers, telemetry and outputs.
The Blueprint ships with connectors and examples: Execution-context connectors for Slurm, Kueue and static inputs; log and metric connectors for Loki, local files, Kubernetes pod logs and PromQL metrics; output connectors for stdout and SQLite. Blueprints can run near each cluster’s data sources with either a dedicated Attribution Service per Blueprint or a shared Service for multiple Blueprints. Downstream systems consume terminal_state and sub_domain instead of parsing product-specific logs.
Extending Attribution to your infrastructure
Open-sourcing NVIDIA Attribution allows teams to encode failure patterns once and apply them across workloads and clusters. Developers can add rules for new failure modes, implement connectors for their schedulers and telemetry backends, and route AttributionBlueprintResult records into incident, recovery and fleet-management systems. New integrations use the same ExecutionContext, Observation and result contracts, so they feed the same diagnosis pipeline.
Walkthrough and quickstart
The article provides a walkthrough that analyzes NVLink error evidence from a recorded training workload. The sample uses two components: the Attribution Service (runs analysis rules) and the Attribution Blueprint (reads workload context and logs, calls the Service, and stores results for inspection). Sample evidence is included in the repository so you can follow the analysis before connecting a live cluster.
Quickstart commands (repository clone and example run) are provided to build a Docker image for the Blueprint dashboard, run it locally, and invoke the Blueprint against the local Attribution Service. After running the demo, open http://127.0.0.1:8080 to inspect each workload’s diagnosis, affected nodes, supporting symptoms and evidence trail.
Rules and testing
The rules engine uses checked-in declarative and Python rules. A production rule example recognizes an invalid Pyxis argument in an application log and includes a co-located test case. On startup the Service loads the bundled rule catalog and evaluates inline cases. Rule changes are committed and shipped with the Service; rules are not uploaded dynamically through the HTTP API. The repository includes a rule-authoring guide describing how to add new extractors, register symptoms, add tests and verify the catalog.
Get started
Clone the NVIDIA Attribution repository and follow the quickstart to run the sample and inspect diagnoses. For production use, consult the Blueprint configuration reference for scheduler and telemetry connectors and the documentation on adding extractor rules for new evidence patterns.
Acknowledgements
Aaron Blakeman, Ajay Hotchandani, Alex Shaposhnikov, Ashwin Poojary, Ben Gueret, Chunyang Sheng, Dorrin Poorkay, Gargi Prasad, Felix Abecassis, Grace Tan, Jason Sewall, Mayuri Gupta, Pasha Shamis, Pavan Kumar Gondhi, Piero Orderique, Rahul Kandu, Rakesh Konda and Sirshak Das contributed to NVIDIA Attribution and to this post.
