NVIDIA AI Cluster Runtime (AICR), a modular open-source component of the NVIDIA DSX AI Factory Platform, now includes a qualified AI bill of materials (AIBOM) integration that uses Google’s k8s-aibom to inventory AI workloads running on Kubernetes. k8s-aibom observes workload and pod APIs and emits CycloneDX 1.6 AIBOMs for recognized AI workloads.
Why this matters
Platform teams use AICR to produce known-good, version-locked GPU platform recipes, but application teams define and change workloads independently of those recipes. A reproducible deployment bundle therefore cannot by itself prove which workload artifacts are actually running on each node. Mutable image tags like :latest add risk: identical tags can point to different images over time, and a node that pulls later may retrieve an image not covered by an earlier security review.
What AICR and k8s-aibom do together
- AICR converts validated GPU platform configurations into version-locked recipes and reproducible deployment bundles. Each recipe records components, version pins, values, constraints, deployment ordering, and validation expectations. Bundles include checksums; signing is optional via aicr bundle --attest.
- k8s-aibom watches Kubernetes workload and pod APIs, and for each recognized AI workload emits an AIBOM. Coverage is limited to supported workload kinds that carry a recognized AI signal (native Dynamo graphs, NIMService, LeaderWorkerSet, DisaggregatedSet, CronJob); tracked child workloads are combined into their owner’s AIBOM.
Together they close the gap between declared platform intent and observed runtime assets: AICR supplies a validated, reproducible GPU platform configuration; k8s-aibom inventories supported AI workloads and records how confidently each attribute was determined. This combination helps find unregistered or shadow workloads, identify models and runtimes, track external AI APIs and vector databases, detect drift in dependencies, and support security reviews and audits.
Scope and limitations
AICR operates at the runtime deployment layer (drivers, operators, kernels, Kubernetes components, deployment configuration). k8s-aibom operates higher in the stack and later in the cluster lifecycle, observing what workloads declare and what pod evidence is available. Neither view alone is sufficient for complete AI governance; the accountability gap lies between them.
Usage example and required versions
The examples in this article reflect AICR v1.2.0 and a catalog that pins k8s-aibom 1.6.2. Use a CLI build that matches the catalog to reproduce the examples. The workflow assumes a Google Kubernetes Engine (GKE) cluster running Kubernetes 1.32 or later, NVIDIA H100 GPU nodes, Container-Optimized OS (COS), and GKE-managed drivers and device plugins. The sequence shown generates a recipe, renders it into a bundle, deploys it, and validates the result without manual assembly.
Generate the recipe
Example command to create a recipe:
aicr recipe \
--service gke \
--accelerator h100 \
--os cos \
--intent inference \
--output recipe.yaml
k8s-aibom is selected as a component in the recipe; you can query the resolved value without writing a file:
aicr query \
--service gke --accelerator h100 --os cos --intent inference \
--selector components.k8s-aibom.version
# 1.6.2
Bundle and deploy
Create a deployment bundle (adjust node selectors to your environment):
aicr bundle \
--recipe recipe.yaml \
--system-node-selector nodeGroup=system-pool \
--accelerated-node-selector nodeGroup=gpu-worker \
--output ./bundles
A bundle is one directory per component, ordered by the dependency graph, each carrying its values, checksums, and an install script. AICR supports multiple deployers via the --deployer flag (helmfile, flux, argocd, etc.). Verify and install:
(cd ./bundles && aicr verify . && chmod +x deploy.sh && ./deploy.sh)
After deployment the k8s-aibom controller runs in the k8s-aibom-system namespace. AICR uses the main chart’s CRDs; check the installed CRD storage version:
kubectl get crd aiboms.aibom.k8saibom.dev \
-o jsonpath='{.spec.versions[?(@.storage==true)].name}'
# v1beta1
Validate
Run snapshot and validation steps:
aicr snapshot --output snapshot.yaml
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml \
--phase deployment --phase conformance
The k8s-aibom health check runs during the deployment phase and requires the controller to be available, AIBOMControllerConfig Ready=True, and CRDs to report the storage version expected by the pinned chart.
Opt in a namespace and run a workload
Inventory is namespace opt-in. Label the namespace to enable AIBOM emission:
kubectl create namespace prod-jobs
kubectl label namespace prod-jobs aibom.k8saibom.dev/enabled=true
A simple vLLM deployment serving Qwen (note the image uses :latest, a common pattern in some environments):
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-qwen
namespace: prod-jobs
spec:
replicas: 1
selector:
matchLabels: {app: vllm-qwen}
template:
metadata:
labels: {app: vllm-qwen}
spec:
tolerations:
- key: dedicated
operator: Equal
value: gpu-workload
effect: NoSchedule
nodeSelector:
cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb
containers:
- name: vllm
image: vllm/vllm-openai:latest
args: ["--model", "Qwen/Qwen2.5-0.5B-Instruct",
"--max-model-len", "4096",
"--gpu-memory-utilization", "0.30"]
ports:
- containerPort: 8000
resources:
limits:
nvidia.com/gpu: "1"
volumeMounts:
- {name: shm, mountPath: /dev/shm}
volumes:
- name: shm
emptyDir: {medium: Memory, sizeLimit: 2Gi}
Apply the manifest:
kubectl apply -f vllm-qwen.yaml
View the inventory:
kubectl -n prod-jobs get aiboms.aibom.k8saibom.dev
Example line from the controller:
NAME WORKLOAD NAME CATEGORY RUNTIME READY AGE apps-deployment-vllm-qwen Deployment vllm-qwen inference vllm True 28m
The AIBOM is constructed from the workload spec, so it can exist before any pod is running. An AIBOM may report Ready=True while the pod is still pulling its image; it reports declared and resolved attributes but does not prove successful scheduling or execution.
Verify the AIBOM
The AIBOM document can be stored inline and carries a published SHA-256 over its canonical bytes. Example commands to extract and inspect the BOM:
kubectl -n prod-jobs get aibom apps-deployment-vllm-qwen -o json > aibom.json
jq -r '.status.bomDocument.inline.data' aibom.json | base64 -d > bom.json
jq -r '.status.bomDocument.sha256' aibom.json
shasum -a 256 bom.json | awk '{print $1}'
jq -r '.components[] | "\(.type)\t\(.name)"' bom.json
Typical entries include application vllm, container vllm/vllm-openai, machine-learning-model Qwen/Qwen2.5-0.5B-Instruct. Each attribute includes one of four confidence tiers: declared, inferred, unresolved, or verified, plus an evidence locator.
Resource impact observed
Upstream measurements of k8s-aibom v1.6.1 on GKE with NVIDIA Dynamo and the AICR configuration reported controller inventory overhead as follows:
- ~4 API-server requests per minute during a 15-minute steady-state window
- Average controller CPU usage: 2.2 millicores
- Controller memory working set: 28–34 MiB
- One AIBOM regeneration when the tested workload’s image tag changed
What remains to be done
The integration deliberately omits or leaves disabled certain features that would enable stronger guarantees:
- Signed runtime observations: the AIBOM resource itself is not signed; signing would make observations tamper-evident (canonical bytes and SHA-256 make signing tractable).
- Connecting verified provenance: k8s-aibom can verify Sigstore signatures on model claims and assign a verified confidence tier, but AICR leaves that verification disabled by default. AICR can produce transparency-logged bundle attestations; integrating that platform provenance with runtime inventory is future work.
- Correlating intent and observed workloads: recipes express deployment intent; runtime AIBOMs record observed specs and pod evidence. Correlating them requires shared artifact identities, freshness rules, and mapping between infrastructure and workload coverage — that comparison remains to be implemented.
Where to find the projects
- NVIDIA AI Cluster Runtime (AICR): NVIDIA/aicr (open source)
- k8s-aibom: GoogleCloudPlatform/k8s-aibom (open source)
For teams running GPU workloads on Kubernetes, adding k8s-aibom via AICR provides an inventory of recognized AI workloads plus evidence about how each attribute was determined.
