AI infrastructure spans multiple layers: platform hardware (PSUs, fans, BMCs), GPUs, network and rack‑level interconnects, cluster/job management, and application/inference services. When performance drops, symptoms often appear in one layer while the root cause lies in another. Full‑stack observability connects telemetry across these layers so operations teams can detect problems, isolate causes, and keep AI workloads reliable.
This article gives a practical approach for NVIDIA AI infrastructure: enumerate failure domains that must be observable, map components to telemetry tools using a decision framework based on NVIDIA DGX deployments, and reduce telemetry to a short, actionable alert set and a single triage dashboard.
A common failure mode: gray failures and cascading slowdown
Consider a distributed training job three days into execution. GPU utilization and queue wait times look normal, yet throughput is reduced for six hours. Investigation traces the problem to one InfiniBand link with an elevated bit error rate (BER). This is a classic gray failure: the hardware is degraded but not ‘‘down.’’
In Bulk Synchronous Parallel (BSP) style training, tightly coupled ranks are sensitive to stragglers: one slow rank slows the whole job. Link‑level retransmits can stall a single rank during synchronous collective operations like NVIDIA Collective Communications Library (NCCL) all‑reduce. The effective throughput becomes the slowest rank’s throughput and the other ranks block, producing a cascading failure that can waste large amounts of GPU time.
What domains must be observable?
Before selecting monitoring software, list the domains whose silent failures consume GPU hours:
- Platform health: fans, PSUs, BMC, chassis, CPU, memory, local storage.
- GPU health and performance: utilization, temperature, power, XID/ECC events, NVLink throughput.
- Fabric: InfiniBand or Ethernet link integrity, congestion, switch/cable health; rack‑scale NVLink where present.
- Cluster and jobs: scheduling, reservations, idle allocated GPUs, queue wait times.
- Inference services: latency, success rate, cache behavior when NVIDIA NIM microservices or similar are in production.
Coverage gaps often reveal themselves under load; analyzing latent failure modes early reduces discovery time and helps prevent recurrence during scale runs.
Map components to telemetry tools: a decision framework
Operations teams need a clear mapping from component to telemetry source. A decision framework derived from DGX deployments maps Redfish/IPMI, NVIDIA Data Center GPU Manager (DCGM), NVIDIA System Management (NVSM), NVIDIA Unified Fabric Manager (UFM), NVIDIA NetQ, NVIDIA NMX, NVIDIA Base Command Manager (BCM), and NVIDIA Run:ai to infrastructure domains. The practical guidance is:
- Redfish/IPMI: platform node hardware (fans, PSUs, BMC) coverage.
- DCGM: preferred for GPU metrics (utilization, power, temperature, NVLink, XID/ECC) and Prometheus export.
- NVSM: system health aggregation on DGX‑class nodes (drives, power, overall health).
- UFM: InfiniBand fabric visibility (port health, BER, congestion, routing).
- NetQ: Ethernet/RoCE fabric visibility.
- NMX: required for rack‑scale NVLink telemetry; not necessary for classic multi‑node NVLink covered by DCGM.
- BCM: cluster/job aggregation plane; not the primary source for low‑level counters.
- Run:ai and NIM: introduce when workload scheduling fairness or inference SLOs are operational priorities.
Key tradeoffs:
- DCGM vs NVSM for GPUs: use DCGM for per‑GPU telemetry into Prometheus; retain NVSM for system‑level DGX health. Neither replaces BMC/Redfish data.
- UFM vs NetQ: choose by fabric type—InfiniBand uses UFM; Spectrum Ethernet/RoCE uses NetQ.
- NMX: include only for rack‑NVLink; otherwise DCGM covers node interconnects.
- BCM: use as aggregator and job plane, not as the sole source of telemetry.
A useful rule: cover every required full‑support domain with the fewest tools. Extra exporters without a clear triage path add noise and cause alert fatigue.
Applying the framework: InfiniBand DGX cluster example
Example environment: DGX cluster with InfiniBand, BCM and Slurm; mostly training jobs, no inference in production. Operational requirement: a single triage dashboard and alerts that detect fabric and GPU regressions before multi‑hour job waste occurs.
Decision process:
- In‑scope domains: platform, GPU, InfiniBand fabric, cluster/jobs. Out of scope initially: NetQ, NMX, Run:ai, NIM.
- Tool selection: Redfish/IPMI on every node (fans, PSU, BMC state); DCGM on every GPU node (utilization, power, temp, XID/ECC, NVLink); NVSM on DGX nodes (system health aggregation); UFM for InfiniBand port health, BER, congestion and routing; BCM as cluster aggregator for jobs, reservations and consolidated hardware alerts.
- Rationale: DCGM alone would miss BER regressions that cause tail‑at‑scale problems. UFM alone would miss GPU XID storms or node power faults. BCM alone does not provide low‑level counters. Together they cover the domains that waste GPU hours.
- Exclusions: skip NetQ, NMX and inference metrics until Ethernet, rack‑NVLink or inference services are introduced.
Initial stack: Redfish/IPMI, DCGM, NVSM, UFM and BCM, unified into Prometheus/Grafana.
Build a short, actionable alert set
Most tools expose hundreds of metrics. Prefer a short top‑k set tied to service‑level indicators (SLIs) and service‑level objectives (SLOs). Each alert should map to a clear remediation action. Start with documented high‑frequency telemetry fields rather than all counters.
Starter metrics for an InfiniBand cluster:
- Platform (Redfish/IPMI): fan speed (SPD_FAN_), PSU status (PWR_), key temperatures (TEMP_*).
- GPU (DCGM, NVSM): DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_MEM_COPY_UTIL, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_XID_ERRORS, plus NVSM GPU/system health alerts.
- InfiniBand (UFM Telemetry): PortXmitDataExtended, SymbolErrorCounterExtended, Effective_BER, Total_Raw_BER, Chip_Temp.
- Jobs (BCM/Slurm): running jobs, GPU reservations, wait time so fabric or GPU alerts can be correlated with workload impact.
Expand the set only when incidents show coverage gaps. Alert on symptoms that correspond to defined playbook actions (drain node, replace cable, open a fabric case), not every counter.
Use Prometheus exporters where available: DCGM and NVSM expose Prometheus endpoints; UFM and BCM can feed the same scrape model via exporters or APIs, keeping protocol selection simple for an initial deployment.
Build a unified triage dashboard
With tools and top‑k metrics selected, create a unified AI infrastructure triage dashboard:
- Install IPMI and DCGM exporters on every GPU node.
- Run UFM Telemetry where the fabric is reachable; ingest it into Prometheus.
- Retain BCM as the cluster management and aggregation plane.
- Point Grafana at Prometheus for dashboards and alerts across GPU, node and fabric signals.
- Add a community Slurm dashboard if job‑level context is not already available in BCM.
Use a two‑layer monitoring approach: Layer 1 is a high‑level Grafana triage board that answers ‘‘is the fault in GPU, node or fabric?’’. Layer 2 are vendor UIs (UFM web UI, BCM Base View, etc.) for deep root‑cause analysis once the failing domain is known.
Observability acceptance criteria and expansion
You are ready to expand when:
- Every in‑scope failure domain has at least one full‑support tool from the framework.
- Alerts are bound to a short top‑k metric list with owners and actions.
- GPU, node and fabric signals share a common timeline in a single view.
- Additional tools (Ethernet, rack‑NVLink, Run:ai, NIM) are added only when required, not by default.
Measure observability maturity by whether signals identify the failing component and the next action before significant compute capacity is wasted, not by the number of dashboards.
Recommended expansion order: enable DCGM GPU telemetry, validate DGX system health via NVSM, configure fabric visibility with UFM or NetQ (depending on fabric), adopt BCM for cluster aggregation and operations, and add NVIDIA NIM Operator for inference observability when needed. Keep metric dictionaries and protocol matrices in runbooks or product docs and keep the production alert set small enough for on‑call use.
3‑step rollout checklist
- Establish coverage: pick one full‑support tool for each in‑scope domain.
- Integrate exporters: wire Redfish/IPMI, DCGM, NVSM, UFM and BCM into Prometheus and point Grafana at the unified telemetry endpoint.
- Enforce ownership: bind every alert to an owner and a playbook action before adding the next exporter.
A decision framework leads to a few well‑chosen signals and a single triage board instead of fifty unread dashboards. That focused approach protects GPU hours from silent, recurring failures.
