AI factories — both cloud and on‑premises clusters that run large AI training and inference workloads — are power‑limited systems where efficient GPU placement matters. Poor placement fragments topology domains and forces traffic over shared links and switches, reducing throughput, increasing job cost, and leaving provisioned GPUs idle while waiting for data.
Because GPUs exchange data continuously during training and inference, distributed workloads gain from communication locality: colocated GPUs see much shorter, higher‑bandwidth paths than GPUs separated across topology domains. NVIDIA NVLink and NVLink Switch provide high‑bandwidth, all‑to‑all scale‑up connectivity inside rack‑scale GPU domains, while NVIDIA Spectrum‑X Ethernet and Quantum InfiniBand provide predictable, low‑latency scale‑out networking.
A scheduler can only make good placement decisions when it has a current, accurate view of GPU and fabric relationships. Keeping that view up to date as clusters change is the practical failure point Topograph aims to solve.
What Topograph does
NVIDIA Topograph is an open‑source toolkit that discovers a cluster’s network and accelerator topology from cloud APIs or on‑premises fabric systems, normalizes the data into a canonical model, and publishes outputs in the formats workload managers expect. It supports Kubernetes node labels, Slurm topology configuration (tree/block and per‑partition YAML), Slinky ConfigMaps, Node Feature Discovery (NFD) resources, and instance‑oriented topology JSON.
Inside the NVIDIA DSX OS orchestration layer, Topograph integrates with Dynamic Resource Allocation (DRA) and KAI Scheduler to enable topology‑aware gang scheduling across AI factory infrastructure.
Why topology awareness matters (numbers and examples)
Topology can be the difference between a local, nearly non‑blocking path and a cross‑domain flow that competes for shared bandwidth. For reference from NVIDIA: modern Quantum InfiniBand ports can reach up to 800 Gb/s; NVIDIA NVLink in its fifth generation (Blackwell GPUs such as GB200/GB300) delivers 1.8 TB/s bidirectional bandwidth per GPU, and sixth generation (Vera Rubin) delivers 3.6 TB/s per GPU through a dedicated NVLink Switch fabric. That design gives each GPU its own high‑capacity lane instead of forcing heavy sharing under load.
Schedulers that operate from a current topology view can prefer GPUs within the same topology domain and avoid placing tightly coupled tasks across distant domains where contention and latency increase.
Architecture: providers and engines
Topograph’s model separates providers and engines. A provider discovers topology from cloud provider APIs or fabric systems and converts it into a canonical representation. An engine translates that canonical model into the scheduler‑specific outputs:
- Kubernetes node labels
- NFD NodeFeature/NodeFeatureGroup objects
- Slurm tree/block files or per‑partition YAML
- Slinky ConfigMaps
- Instance‑oriented topology JSON
Cloud integrations with working Topograph providers include Google Cloud, Lambda, Nebius, Nscale, and Oracle Cloud Infrastructure (OCI); additional cloud and colocation providers are in development. On‑premises deployments typically use the InfiniBand provider with ibnetdiscover, or NetQ for Spectrum‑X or Multi‑Node NVLink (MNNVL) domains. The provider interface is open so operators can implement and contribute providers for their environments.
A note on freshness: Topograph maintains a current topology view with five cooperating components — API Server, Node Observer, Node Data Broker, Provider, and Engine — that validate requests, watch for node/Pod changes, collect per‑node attributes, convert provider data to the canonical model, and write scheduler‑ready outputs.
API and operational details
Topograph exposes five main endpoints:
- POST /v1/generate — submit an asynchronous generation request, returns a request ID with HTTP 202
- GET /v1/topology?uid=<request-id> — polling endpoint: 202 while processing, 200 with result when complete
- POST /v1/lookup — return cached status/result without re‑submitting work
- GET /healthz — liveness
- GET /metrics — Prometheus metrics
The project uses a short aggregation delay (15 seconds is typical) so repeated identical requests within that window are coalesced and processed once. For testing without physical hardware, Topograph supports simulation models and helpers (kwok‑nodes, Kind/KWOK) to create virtual nodes.
Kubernetes usage (engine: k8s)
The default Kubernetes scheduler does not discover physical interconnect hierarchy. Topograph fills that gap by publishing provider‑reported topology as node labels. Prerequisites include Kubernetes 1.27 or later, Helm 3.10+ (or Helm 4.x), kubectl permissions, and a supported provider. KAI Scheduler or Kueue TAS are optional components for topology‑aware gang scheduling.
Example Helm install:
helm repo add topograph https://dsx-ai-factory.github.io/topograph helm repo update helm install topograph topograph/topograph --namespace topograph --create-namespace --set engine.name=k8s --set provider.name=<provider>
Topograph labels fabric locality and accelerator domains with keys such as:
- fabric.topograph.run/tier-0 (leaf switch closest to node), tier-1, tier-N
- accelerator.topograph.run/domain
- accelerator.topograph.run/sub-domain
The Kubernetes engine publishes only the tiers discovered; operators can customize label keys with engine parameters. Labels are visible with kubectl get nodes --show-labels | grep -E 'fabric.topograph.run|accelerator.topograph.run'. The default ClusterIP API address is topograph.topograph.svc.cluster.local:49021; for local debugging you can port‑forward to that port and call /healthz.
These node labels can be used as topologyKey values in Pod affinity rules. For strict gang placement, KAI Scheduler or Kueue can consume the same labels and enforce multi‑Pod locality. KAI Scheduler exposes a Topology resource that organizes node labels into hierarchy levels; a Job can be annotated to require placement inside a tier‑1 domain while preferring concentration in a tier‑0 domain when possible.
Topograph also supports publishing topology through the NFD engine. The nfd engine emits NodeFeature and NodeFeatureGroup objects; this requires enabling the alpha NodeFeatureGroupAPI feature gate. NFD output is useful for downstream components that already consume NodeFeatureGroup objects but is not a substitute for using Kubernetes topologyKey labels when native Pod affinity is required.
Slurm usage (engine: slurm)
Topograph can generate Slurm configurations in tree and block formats, and it supports Slurm’s per‑partition YAML configuration introduced in Slurm 25.05. Typical Slurm deployments install Topograph as a native package (Debian or RPM). The service configuration points to port 49021 and the chosen provider and engine.
To trigger discovery you POST to /v1/generate and poll /v1/topology for the generated configuration. Engines can write files directly (for example /etc/slurm/topology.conf or /etc/slurm/topology.yaml) and optionally run scontrol reconfigure. Topograph also supplies helper scripts to register node up/down triggers for refreshing topology when Slurm node states change.
Slinky usage (engine: slinky)
Slinky (SchedMD’s project for running Slurm on Kubernetes; SchedMD was acquired by NVIDIA in December 2025) is supported: the Slinky engine maps Kubernetes nodes to slurmd Pods and writes Slurm topology data into a ConfigMap. The engine supports cluster‑wide tree/block outputs and per‑partition YAML. For MNNVL deployments, a dra provider reads existing nvidia.com/gpu.clique labels when generating block topologies. Topograph updates the ConfigMap as slurmd Pods change and can also annotate Kubernetes nodes for dynamic Slurm nodes if configured.
When to adopt Topograph
Placement problems scale poorly: suboptimal placement shows up as network congestion and increased cost. Topograph provides schedulers with a current, provider‑reported map of the physical network and accelerator domains so topology‑aware decisions are consistent across cloud and on‑premises environments without manual snapshot maintenance.
Used with KAI Scheduler, Kueue, or native Kubernetes affinity, the topology map can improve AI factory efficiency, tokens per watt, and overall cost for distributed training and inference workloads.
Get started by deploying Topograph from the dsx‑ai‑factory/topograph GitHub repository and consult the DSX OS documentation for integration details.
