NVIDIA has introduced Cosmos, a platform combining world foundation models, development tools and NIM microservices aimed at physical AI. The platform is designed to help adapt world models to sensors, tasks and data, and to deploy those models reliably with GPU‑accelerated serving.
What Cosmos Framework provides
Cosmos Framework is an open, fully modular framework for training, optimizing and running inference with world models. It supplies agent skills and recipe‑style workflows for multimodal data (text, images, video, audio, actions), model training, optimization and inference. In addition to Cosmos3, the Framework is expanding support to models such as NVIDIA Alpamayo, GR00T, Qwen, pi0.5 and other Hugging Face‑compatible checkpoints.
Key capabilities:
- Faster training across text, image, video, audio and action modalities; use LoRA for parameter‑efficient adaptation or full supervised fine‑tuning (SFT) to update all parameters.
- Data ingestion via JSONL, WebDataset, LeRobot or custom adapters; train across GPUs and save checkpoints to resume runs.
- Post‑training skills to accelerate and automate follow‑on work.
Reinforcement learning support
Cosmos Framework covers both online and offline RL. Newly supported pipelines include:
- DPO: offline preference optimization
- GRPO: online RL with rule or model reward and vLLM rollouts
- PPO: actor‑critic RL with frozen reference and critic, including RL for embodied policies in simulation
- OPD: on‑policy distillation from a frozen teacher
- DiffusionNFT: online RL for diffusion world models
Rollout, reward and training roles can run on shared GPUs or separate pools, and supported simulators can be connected to collect experience.
Model optimization and inference
The platform supports model distillation and quantization to reduce size and memory use. Compatible checkpoints can run with Hugging Face Diffusers, PyTorch, SGLang, vLLM‑Omni, vLLM, TRT‑LLM, TRT‑Edge‑LLM and NVIDIA NIM. Cosmos provides evaluation tooling for quality, accuracy and GPU utilization to help maximize compute efficiency.
Using agents and skills
Load setup, inference, post‑training and troubleshooting skills from .agents/skills. A recommended bounded task workflow example: "Post‑train <checkpoint> on <dataset> using <recipe>" — pin revisions, validate datasets, keep training and evaluation episodes separate, run a smoke test, save configuration, stop after N GPU‑hours, and export results. Compare against the base model on held‑out data with three seeds and repeat evaluation at a supported lower precision while recording quality, peak memory and latency.
Which simulation workflows are available?
The Framework provides post‑training and inference recipes for robot simulation, realistic scene generation, multi‑camera driving simulation and surgical policy evaluation. New Cosmos3 Nano simulation recipes and example checkpoints let developers run agent‑runnable skills to evaluate example checkpoints and adapt Cosmos3 Nano to custom sensors, action spaces and domains.
Checkpoints support PyTorch and vLLM‑Omni and were tested on NVIDIA Hopper and Blackwell GPUs. The recipes help select GPU count, memory configuration and inference settings.
Autonomous vehicle (AV) simulation
AV checkpoints accept a prompt and a World Scenario Map (WSM) control that encodes lanes, objects, road markings, traffic signs and road boundaries. Joint RGB–LiDAR generation uses HD‑map rangemap controls and can take optional initial LiDAR observations. Prompts describe weather, lighting, time of day and road conditions, and can include camera‑rig and field‑of‑view context; optional initial images provide visual context for the scene’s appearance.
-
Cosmos3‑Nano‑Transfer‑Auto: a bidirectional generator that denoises a complete multi‑view clip at once for offline synthetic data generation. It supports 30 fps input/output and up to 400 frames at 480p or 720p; longer sequences can be produced by autoregressive chunking. A reference run produces seven 720p videos (297 frames at 30 FPS) and 99 LiDAR sweeps.
-
Cosmos3‑Nano‑Sim‑Auto: a causal, few‑step‑distilled variant that enables autoregressive streaming generation. Observations are generated incrementally as an AV policy or human driving input updates the scenario via a WSM renderer, enabling closed‑loop evaluation where generated camera and LiDAR observations feed back to the policy.
The repository contains published sample checkpoints, input configuration examples and inference guides to reproduce the reference workflows and validate outputs (sample_outputs.json, lidar.safetensors, WSM alignment checks).
Robotics simulation
- Cosmos3‑Nano‑Sim‑Transfer converts depth video plus a prompt into RGB video while preserving geometry and motion, but varying lighting, textures and backgrounds — useful for robot demos and viewpoint changes while retaining actions and states.
- Cosmos3‑Nano‑Sim‑Bimanual generates the next visual observation from the scene and robot actions, supporting interactive two‑arm simulation for YAM and AgiBot embodiments driven by a policy or human joysticks.
Sim‑Transfer consumes a physics simulator’s updated state to create visuals; Sim‑Bimanual generates the next observation directly from scene state and robot actions.
Surgical simulation
Surgical environments are particularly hard for classical simulators because of complex multi‑physics interactions. Generative world models offer a complementary approach by learning visual dynamics from synchronized surgical video and robot kinematics rather than explicitly modeling all physics.
- Cosmos3‑H‑Embodied‑Nano (post‑trained on Open‑H‑Embodiment) can generate surgical video from robot actions, predict actions from video or generate both together, using an initial RGB frame, a task prompt, the last 16 executed actions and subsequent robot actions.
- Cosmos3‑H‑Dreams: a distilled causal few‑step model derived from Cosmos3‑H‑Embodied‑Nano for real‑time closed‑loop simulation; it uses live robot kinematics to generate frames and lets developers interactively explore maneuvers and evaluate policies before testing on hardware.
These models support synthetic data generation and offline policy evaluation without constructing detailed physics models of the surgical scene.
When to use Cosmos NIM for accelerated inference
Cosmos NIM packages provide optimized containers and models for accelerated inference and easier deployment:
- Faster inference and lower memory: techniques include quantization, speculative decoding, caching, token pruning and KV‑cache‑aware routing. Speculative decoding produced measured mean speedups of roughly 2–3× on H200 in NVIDIA’s tests.
- Custom models and live video understanding: serve fine‑tuned Cosmos checkpoints and apply streaming VLM inference for live streams, enabling sub‑second alerts and up to 10× lower latency for faster event response.
- Simpler production deployment: automatic configuration selection, validated performance and quality across supported GPUs, licensing support and quarterly updates.
GPU memory guidance (excerpt)
The documentation provides per‑model VRAM requirements (GiB) by capability and precision. Examples from the released table include:
- Reasoning: Cosmos3‑Edge — 16 GiB BF16; Cosmos3‑Nano — 23.1 GiB BF16; Cosmos3‑Super — 135 GiB BF16
- Generation: Cosmos3‑Nano — 58 GiB BF16 (other precisions and offloading options are listed in the full table); Cosmos3‑Super — 150 GiB BF16
Refer to the full table for exact values per precision and offloading configuration.
Launching Cosmos NIM — high level steps
Choose a GPU and model configuration from the guidance table, install the required NVIDIA driver, Docker and NVIDIA Container Toolkit on Linux, sign in to NGC to generate a personal API key, and export it as NGC_API_KEY in your shell. The published example shows how to log into nvcr.io, create a local cache directory and run a Nano Generator container bound to GPU 0 with appropriate shm, ulimit and port settings. After a few minutes the service becomes ready; readiness and metadata endpoints (/v1/health/ready, /v1/metadata) let you verify the selected model and profile.
Set NIM_PERF_PROFILE at container launch to choose latency (interactive, GPUs used together per request) or throughput (dataset generation or many concurrent users, independent replicas). Latency profiles support 1, 2, 4 or 8 GPUs; throughput requires 2, 4 or 8 compatible GPUs.
Next steps and resources
Developers should explore Cosmos Framework, reproduce a matching simulation checkpoint workflow, then adapt it to their sensors, actions and datasets. NVIDIA provides a technical report and detailed guides. NVIDIA is hosting a livestream on Thursday, Nov. 5 to demonstrate how to use Cosmos Framework.
Summary
Cosmos bundles a modular development framework, example simulation checkpoints and GPU‑optimized NIM deployment artifacts to support the lifecycle of multimodal world models for AV, robotics and surgical applications. The combination of agent skills, post‑training/inference recipes and NIM containers aims to shorten the path from model research to production deployment with validated performance on 20+ NVIDIA GPUs.
