NVIDIA has created the AI Factory Operations Agent to automate parts of GPU-cluster operations and reduce manual effort by Site Reliability Engineers (SREs). The system, developed to support internal NVIDIA DGX SuperPOD operations, aims to correlate signals across compute, network, hardware and scheduling layers and present findings through a single chat-style interface.
Architecture and how it works
At the core is a user-facing agent runtime that accepts SRE queries via a chat UI and invokes domain-specific sub-agents and tools. Sub-agents communicate through the Model Context Protocol (MCP), an interoperability layer any organization can adopt to connect agents to approved operational tools.
The developer example is built around the NVIDIA NemoClaw blueprint. Agents connect to infrastructure using OpenClaw plugins and MCP servers; each MCP exposes a focused, approved toolset. Users can operate via the built-in web UI or integrate existing code-agent interfaces through an API. This modular design allows adding new agents and infrastructure integrations without altering existing agents.
The framework uses NVIDIA OpenShell and supports fully on-premises deployments, extensibility, and data sovereignty. Agent inference runs on locally served NVIDIA Nemotron open models, and the framework itself runs across NVIDIA DGX H100, NVIDIA DGX B200, and NVIDIA DGX GB300 production clusters.
Kubernetes deployment and runtime details
The AI Factory Operations Agent runs as a Kubernetes control-plane application. Helm deploys the UI, OpenClaw gateway, MCP tool servers, inference endpoint, and NVIDIA OpenShell runtime as Kubernetes services. The chat UI forwards requests to NemoClaw, which routes them to the correct domain agent.
For shell execution, OpenShell creates a per-session Kubernetes sandbox rather than executing commands in the main gateway pod. The sandbox receives the agent workspace, the selected skills, and the command policy; its lifecycle is designed not to modify existing services or production workloads.
Security is enforced in layers. Kubernetes inspection uses a server-side run_kubectl tool that invokes a pinned kubectl binary with registered read-only credentials. OpenShell does not receive kubectl or Kubernetes credentials. Kubernetes service accounts and role-based access control (RBAC) define which cluster resources the tool can inspect. Because checks run through the same Kubernetes access path used by the runtime, the results reflect what the agent can and cannot do inside the cluster.
Sample capability checks executed by the system include:
(agent) $ kubectl auth can-i get pods yes (agent) $ kubectl auth can-i get configmaps yes (agent) $ kubectl auth can-i get secrets no (agent) $ kubectl auth can-i create pods no (agent) $ kubectl auth can-i update pods no
Domain-specific agents and integrated tools
NVIDIA’s internal deployment uses a growing set of data-center domain agents that produce recommendations:
- Job Recovery Agent: detects training job failures and goodput degradation, performs multi-layer root-cause analysis across scheduler, application, hardware, and network layers, and recommends recovery steps once a root cause is confirmed.
- Hardware Agent: triages device-level hardware faults such as PCIe errors, ECC errors, GPU issues, NVMe failures, and thermal anomalies, tracing symptoms to specific components.
- Observability Agent: queries historical Prometheus, telemetry, and integrated Grafana Loki logs to dynamically create dashboards and display them in the web UI.
- Research Agent: searches preloaded documentation including NVIDIA Mission Control manuals and DGX SuperPOD reference architectures; users can also upload runbooks and site-specific guides.
- Kubernetes Agent: provides read-only operational awareness of pods, services, deployments, GPU resources, node status, and scheduling state.
- Cluster Management Agent: interfaces with NVIDIA Base Command Manager APIs through diagnostic MCP tools for advisory cluster management and troubleshooting.
Beyond these domain agents, the framework integrates cross-cutting MCP tools such as web search, code execution, and Grafana dashboards. Organizations can connect custom MCP servers and APIs to adapt the framework to their existing schedulers, observability platforms, cluster managers, and operational procedures.
Lessons learned from production use
NVIDIA measures root-cause analysis accuracy, SRE agreement, and expands the agent ecosystem. Key lessons from production use include:
- Clear agent boundaries: overlapping ownership creates noise rather than signal.
- Read-only first: accurate investigation builds SRE trust before enabling automation.
- Cross-layer correlation: linking workload, hardware, and network signals finds root causes isolated tools miss.
- Precise MCP tools: clear tool definitions improve routing and reduce incorrect tool selection.
- Transparent reasoning: SREs need visibility into tool calls, evidence, and conclusions before acting.
Deployment recommendations and multi-tenant considerations
For large AI factories, NVIDIA recommends a dedicated GPU-accelerated control plane that is isolated from the production cluster. The control plane handles agent orchestration and reasoning while the production cluster supplies telemetry without consuming training GPUs or disrupting workloads.
Internal NVIDIA deployments run model inference on dedicated GPU capacity and use the control-plane CPU Kubernetes admin space for runtime services. Other deployments may choose local or cloud inference endpoints depending on data-sovereignty requirements, cluster size, and available GPU capacity; smaller deployments can run on an RTX PRO Workstation or a DGX Spark.
In multi-tenant AI factories, scoped agents, tools, sandboxes and Kubernetes permissions control what cluster data, tools, and nodes each tenant can access. Tenant SREs see only their namespaces, jobs, and nodes, while central operations retain broader visibility. OpenShell enforces these boundaries by running each tenant’s agents and tools in an isolated sandbox with kernel-level restrictions.
Getting started
The AI Factory Operations Agent developer example and the orchestration framework are available as experimental open source on GitHub. APIs, configuration and deployment requirements may change between releases. NVIDIA customers can contact their account manager to request access to NVIDIA-developed cluster operations agents.
