A central problem in robotics is creating policies that generalize beyond the demonstrations used for training. Policies that work in a training scene often fail when object shapes, positions, or lighting change. Solving this requires understanding the physics behind tasks, not just imitating demonstrations, and that capability depends on the model backbone used.
Why VLM-based VLAs struggle with physical generalization
The standard approach to language-conditioned robot policies attaches an action module to a pretrained vision-language model (VLM), producing a vision-language-action (VLA) model. VLAs have driven progress in generalist manipulation because VLMs provide strong semantic and language understanding.
However, VLMs are trained to describe images and produce text, not to predict how a scene evolves. They do not necessarily learn what happens to a mug when a gripper closes, how a towel folds, or where an object lands when released. As a result, VLAs generalize well semantically but less effectively for physical generalization to unseen behaviors and environments.
What a World Action Model (WAM) changes
A WAM builds the policy on a video world model: the backbone models how the world evolves over time. Post-training therefore does not need to teach dynamics from scratch but specializes a model that already carries a physics prior. NVIDIA’s research summarized in "World Action Models are Zero-shot Policies" and Jim Fan’s "Robotics’ End Game" talk highlights several advantages of WAMs:
- Learns from diverse data: whereas a VLA often needs near-identical demonstrations of the same task, a WAM learns physical dynamics (how objects move when pushed, grasped, or dropped). Diverse interaction data becomes useful training signal and reduces data collection costs.
- Generalizes in the open world: physics is more general than semantics; the way an object falls or slides transfers across objects and scenes the model has never seen.
- Adapts to new robots with few demonstrations: a model that already understands interaction dynamics requires far less task-specific data to specialize to a new arm or gripper.
Practically, these mean less data to reach a capability threshold, more reliable behavior outside the training distribution, and a shorter path to a new embodiment. Because these properties stem from the pretraining backbone, they appear in every policy post-trained from it.
Why Cosmos 3 is a strong foundation
Cosmos 3 is an omni-model world foundation built on a Mixture-of-Transformers (MoT) architecture. Multimodal inputs pass through an autoregressive transformer for symbolic reasoning (producing discrete tokens such as text), which guides a diffusion transformer for continuous modalities including image, video, audio, and action. Text is generated by next-token decoding; other modalities, including actions, are synthesized by iterative denoising. Cosmos 3 is available in three sizes: 4B (NVIDIA Cosmos Edge), 16B (NVIDIA Cosmos Nano), and 64B (NVIDIA Cosmos 3 Super).
The model’s strength comes from its physical-world training data: roughly 767 million images, 348 million videos of real-world dynamics, and 8 million action samples spanning robot manipulation, autonomous driving, camera motion, and egocentric motion.
From world model to robot policy: Cosmos 3 Policy DROID models
Cosmos 3 is the starting point for specialization. Cosmos3-Nano-Policy-DROID is a 16-billion-parameter policy post-trained from Cosmos 3 Nano for the DROID platform, which is a Franka Panda arm with a Robotiq gripper. There is also a 4B version, Cosmos3-Edge-Policy-DROID, post-trained in the same way for on-device deployment.
Given a language instruction and multi-camera observations, these policies generate robot action trajectories. Three properties follow from the omni foundation:
- It imagines while it acts: when the model outputs actions it can simultaneously output a video prediction of what the robot’s cameras would see if those actions were executed. Action and predicted outcome come from the same model at the same time.
- It preserves the omni architecture: post-training does not remove capabilities; the policy checkpoint still reasons and can generate video, not just output joint positions.
- The prior is measurable: the Cosmos 3 technical report compares two DROID policies trained with identical recipes, data, and compute. Starting from an omni checkpoint trained on multi-domain action data raised RoboLab success from 28.1% to 36.8%, indicating the improvement comes from the architecture and pretraining, not only scale.
Deployment considerations
A WAM carries the full generative world model, so it is larger than compact VLAs. Cosmos 3 maps to different deployment tiers instead of forcing a single trade-off:
- Workstation serving (Nano, 16B): Cosmos3-Nano-Policy runs beside the robot rather than on board. In real-world DROID deployment it is served on a single NVIDIA RTX PRO 6000, with the robot streaming observations over the network and receiving action chunks back.
- On-device (Edge, 4B): Cosmos 3 Edge runs the same policy workload directly on embedded hardware. Operating at robot-control resolution (640×360 observations), it generates 32 actions per inference on NVIDIA Jetson Thor and achieves real-time control at 15 Hz. It is supported across NVIDIA edge computers including RTX PRO GPUs, DGX, GeForce RTX GPUs, and Jetson modules such as Jetson T2000 and T3000.
Why build robot policies with Cosmos 3?
- Open foundation and SOTA starting point: the base model, datasets, post-training recipe, trained weights, evaluation tools, and serving stack are released under a license that permits commercial use.
- Faster adaptation: strong physical priors reduce the task-specific data needed. Convert your data to the LeRobotDataset format, then run the published recipe.
- One foundation for many robots: each new embodiment (Franka, dual-arm setups, UR, WidowX) still needs its own post-training, but all start from the same pretrained foundation instead of training a world model from scratch, lowering the demonstration burden for each.
How to get started
The fastest way to evaluate whether a WAM outperforms an existing VLA is to post-train one from Cosmos 3 on your own data and compare. Explore the open Cosmos 3 model collection and datasets on Hugging Face and GitHub, read the Cosmos 3 technical report for full architecture details and evaluations, and tune in to the Cosmos Labs livestream on August 13. Join the Cosmos Discord community for discussion and support.
Conclusion
Moving from VLAs to WAMs shifts focus from learning to act toward learning how the world evolves. Cosmos 3 makes that shift practical by providing a multimodal, physically rich foundation that yields measurable improvements in real-world policy success and reduces the data and time needed to adapt policies to new robots.
