NVIDIA bundles production-ready runtime recipes for running NVIDIA Dynamo on Kubernetes as part of the NVIDIA DSX AI Factory Platform. The NVIDIA AI Cluster Runtime (AICR) packages Dynamo together with the GPU drivers, schedulers, monitoring and ingress components required by a specific environment and workload, pins versions, and runs deployment and conformance checks so platform teams can reproducibly install and validate the runtime before deploying inference workloads.
Why the underlying runtime matters for Dynamo
A DynamoGraphDeployment describes how vLLM, SGLang, or NVIDIA TensorRT LLM should run in cluster pods, but that graph sits on top of a runtime stack. A production inference cluster must ensure the GPU driver stack matches the nodes, GPU nodes are labeled and discoverable, GPUs are exposed via Dynamic Resource Allocation (DRA), the scheduler supports required topology and gang scheduling semantics, node health signals are available, Prometheus collects accelerator and service metrics, and — when Kubernetes-native ingress is desired — Gateway API / Endpoint Picker (EPP) paths are configured. AICR recipes capture these runtime requirements.
What AICR provides for Dynamo
An AICR Dynamo recipe packages the dynamo-platform Helm chart (pinned in the example to 1.4.2), including the Dynamo operator, runtime images, and the nvidia.com/v1beta1 DynamoGraphDeployment CRD (v1beta1 replaces the older v1alpha1 API). The recipe also installs and pins versions for supporting components such as:
- NVIDIA GPU Operator (driver and GPU software stack)
- Node Feature Discovery (GPU discovery and labeling)
- DRA Driver for NVIDIA GPUs (Kubernetes-native GPU allocation)
- KAI Scheduler (topology- and gang-aware placement)
- Grove (orchestration of multi-component serving graphs)
- NVSentinel and Nodewright (node health, where supported)
- Prometheus stack (cluster, accelerator, and Dynamo metrics)
- agentgateway and Gateway API resources (when ingress is Kubernetes-native)
The recipe records pinned component versions, node selectors, tolerations, checksums and validation expectations to make deployments reproducible. With Dynamo 1.4, components discover each other via Kubernetes resources and AICR’s default path no longer installs NATS by default (it remains opt-in).
KV-cache, disaggregated serving and event planes
Agentic workloads extend long token prefixes (system prompts, tools, conversation history). If a request reaches a worker that does not hold the prefix in GPU memory, the system has to reprocess intermediate tokens. Dynamo routes requests to workers that already hold relevant KV blocks, either through its frontend router or via GAIE/EPP integration at the Gateway API layer.
Dynamo supports disaggregated serving where prompt processing (prefill) and token generation (decode) are separated. On KV-cache misses the prefill pool must reprocess some or all of the prompt, then decode workers generate tokens; this lets operators scale prefill and decode capacity independently. The request plane and KV-cache event plane are separate: in the AICR default for Dynamo 1.4.2 TCP is used for requests and ZMQ for KV-cache events. For example, a vLLM worker can publish raw KV-cache events locally with:
--enable-prefix-caching --kv-transfer-config
{"enable_kv_cache_events": true, "publisher": "zmq", "endpoint": "tcp://*:5557"}
The Dynamo worker runtime subscribes to the local vLLM event source, normalizes and republishes events onto the Dynamo event plane; both the native Dynamo Frontend router and the GAIE/EPP path use that Dynamo-managed KV-cache state for routing decisions.
Dynamo routing paths on Kubernetes
Dynamo supports two routing topologies on Kubernetes; AICR ships and validates the runtime for either.
Path 1 — Dynamo Frontend router (default)
The default topology exposes a Dynamo Frontend behind a Kubernetes Service. When run with DYN_ROUTER_MODE=kv, the Frontend tokenizes requests, scores eligible workers by prefix overlap and current load, selects a worker and dispatches the request over the Dynamo request plane (TCP by default). This is the smallest topology and the most production-proven serving option. It suits scenarios where a model-specific endpoint is owned by an application team or when an edge proxy simply forwards traffic to the Dynamo Frontend Service.
Path 2 — Gateway API Inference Extension (GAIE) and Dynamo EPP
This path integrates Dynamo with a Gateway API implementation and GAIE. In the AICR example, agentgateway resolves an HTTPRoute to an InferencePool. The Dynamo operator creates that InferencePool from the DynamoGraphDeployment; the selector identifies eligible worker pods and endpointPickerRef points to the Dynamo EPP Service.
The gateway invokes the Dynamo EPP via GAIE’s endpoint-selection interface. The EPP tokenizes the request and uses Dynamo’s KV router to select an endpoint, returning the selection and routing metadata to the gateway (the actual HTTP proxy). In this mode, each routable worker pod runs a Dynamo Frontend sidecar in direct mode (--router-mode direct) that receives the forwarded request and passes it to the local worker container.
This topology is appropriate when the platform team manages shared Kubernetes ingress and wants gateway-level control over timeouts, retries and fallback. Both routing approaches can operate behind a Gateway API HTTPRoute; the difference is whether the Dynamo Frontend or the Gateway manages transport to the worker.
Kinetics note: Dynamo 1.5 introduces a native Rust EPP that removes the Go/CGO bridge, implements GAIE’s Lightweight Endpoint Picker (ext_proc) directly, runs the full Dynamo router in-process, and reduces per-request EPP overhead.
How to choose between the two paths
Choose the Dynamo Frontend router when you want the simplest, model-specific topology with minimal infra. Choose the GAIE/EPP path when a shared, platform-managed ingress should control worker selection and network-level behaviors. Both use worker-published KV-cache events for cache-aware placement; AICR’s default enables live events rather than predicted cache state, which helps routing quality under memory pressure.
The AICR recipe workflow
AICR separates runtime setup from application deployment. The main steps shown in the walkthrough are:
- Snapshot the cluster
- Example: aicr snapshot --namespace aicr-validation --output snapshot.yaml
- Snapshot captures cluster facts that AICR uses for recipe selection and validation.
- Generate the Dynamo inference recipe
- Example: aicr recipe --snapshot snapshot.yaml --intent inference --platform dynamo --output recipe.yaml
- The generated recipe records pinned versions, environment configuration, compatibility constraints and validation checks.
For example, AICR v1.0.0 Amazon EKS Dynamo inference recipe selects Helm chart versions such as NVIDIA GPU Operator v26.7.1, Node Feature Discovery 0.19.0, cert-manager v1.20.2, NVIDIA DRA GPU Driver 0.5.0, KAI Scheduler v0.16.9, Grove v0.1.0-alpha.13, Dynamo Platform 1.4.2, Agent Gateway v1.5.0, and Kube Prometheus Stack 84.4.0.
- Render the runtime bundle
- Example: aicr bundle --recipe recipe.yaml --accelerated-node-selector nodeGroup=gpu-worker --accelerated-node-toleration dedicated=worker-workload:NoSchedule ... --deployer helmfile --output bundle
- The bundle contains Helm values and manifests rendered for the target cluster, with placement flags for node selectors and tolerations.
- Deploy the runtime
- Use Helmfile (helmfile -f bundle/helmfile.yaml apply) or the generated deploy script for a local test (bundle-helm/deploy.sh). AICR encodes dependency tiers and Helmfile applies them in order.
- Validate the installed runtime
- Example: aicr validate --recipe recipe.yaml --node-selector nodeGroup=gpu-worker --toleration dedicated=worker-workload:NoSchedule --phase deployment --phase conformance --output report.json
- AICR runs deployment and conformance validation Jobs against the live cluster. The DynamoGraphDeployment application is installed only after these checks pass. Performance validation is run separately.
What AICR validation checks
Validation checks vary by recipe; for a Dynamo inference recipe typical checks include:
- Cluster baseline: expected resources, GPU allocatable capacity, working nvidia-smi on accelerator nodes
- Platform health: GPU Operator health and version, and Dynamo operator/platform health
- Accelerator access: DRA support and secure accelerator access
- Scheduling and scaling: gang scheduling, pod autoscaling, cluster autoscaling and controller robustness
- Observability: Prometheus-backed accelerator and AI service metrics
- Ingress: Gateway API and agentgateway readiness for Kubernetes-native ingress
These checks verify the installed runtime against the recipe’s requirements before application deployment.
Getting started
NVIDIA AI Cluster Runtime (AICR) and NVIDIA Dynamo are open-source parts of the NVIDIA DSX AI Factory Platform. Teams deploying Dynamo on Kubernetes can use AICR recipes and examples from the NVIDIA DSX AI Factory GitHub. Community discussion is available in the NVIDIA DSX AI Factory Slack. NVIDIA also offers hands-on workshops and demos (for example at KubeCon + CloudNativeCon North America), and live demos at the NVIDIA booth (#170). Registration is required for some sessions.
Quick command reference:
- Snapshot: aicr snapshot --namespace aicr-validation --output snapshot.yaml
- Generate recipe: aicr recipe --snapshot snapshot.yaml --intent inference --platform dynamo --output recipe.yaml
- Render bundle: aicr bundle --recipe recipe.yaml [placement flags] --deployer helmfile --output bundle
- Deploy: helmfile -f bundle/helmfile.yaml apply (or generated deploy.sh)
- Validate: aicr validate --recipe recipe.yaml --phase deployment --phase conformance --output report.json
AICR provides a reproducible, validated foundation for Dynamo-based inference services on Kubernetes.
