NVIDIA demonstrated an automated agent‑driven workflow that post‑trains the Cosmos 3 Nano vision‑reasoning model with TAO agent skills. Using a coding agent (example: Codex) and two natural language prompts, the pipeline ran a zero‑shot baseline, an automated LoRA adaptation (~30 minutes training on eight NVIDIA A100 GPUs), and a TAO AutoML hyperparameter sweep, ultimately reaching 93.35% validation accuracy on a traffic safety video QA task.
Problem and significance
Adapting vision‑language models to production video tasks often consumes days of engineering: data formatting, container setup, training scripts, baseline evaluation, and hyperparameter sweeps. NVIDIA’s approach uses the Cosmos 3 Nano foundation model together with NVIDIA TAO agent skills to automate these steps and compress the effort into an automated, one‑day workflow.
Models and architecture
NVIDIA Cosmos 3 is an omnimodal mixture‑of‑transformers (MoT) foundation model family that unifies text, image, video, ambient sound, and action tracking. The MoT layer separates functionality into two towers: an autoregressive Reasoner for multimodal understanding and logic, and a diffusion‑style Generator for predicting future states and actions. Cosmos 3 is available in multiple sizes, including Nano 16B and Super 64B.
Post‑training approaches: SFT vs. LoRA
Two common strategies for adapting foundation models are:
- Full‑Parameter Supervised Fine‑Tuning (SFT): updates all model weights; appropriate for large domain shifts but computationally expensive and can risk degrading general knowledge.
- Low‑Rank Adaptation (LoRA): freezes base weights and injects trainable low‑rank matrices; enables fast iteration.
In the presented experiments, LoRA required approximately 7× fewer GPU hours than full‑parameter SFT for Cosmos 3 Nano, making a one‑day post‑training turnaround feasible.
Two ways to run post‑training
- Open framework: the fully open Cosmos 3 post‑training framework exposes training recipes, dataset formats, and configuration files so teams can build custom pipelines.
- TAO agent skills: these sit on top of the framework and automate the pipeline. A coding agent uses TAO skills to reason about and execute data fixes, container launches, training, and AutoML sweeps.
Installation and prerequisites
TAO skill sets can be installed for supported coding agents; the example uses Codex. A fast install script from the TAO skills bank is provided:
curl -fsSL https://raw.githubusercontent.com/NVIDIA-TAO/tao-skills-bank/main/scripts/install-codex-agents.sh | bash
Required environment variables for the experiment:
export HUGGINGFACE_TOKEN="your_hf_token" export NGC_API_KEY="your_ngc_key" export AUTOML_LLM_API_KEY="your_llm_key"
Dataset and task
The demonstration uses the Toyota Woven Traffic Safety (WTS) dataset for a four‑way multiple‑choice video question‑answering task. The dataset includes more than 8,000 training and validation samples and requires detailed scene understanding of roads, vehicles, signals and pedestrians.
Sample question from the dataset:
- Question: What is the formation of the road?
- Options: A: Single road (right curve), B: Single road (straight line), C: Intersection (without signal), D: Intersection (with signal)
- Expected answer: D
Running a LoRA post‑training with one prompt
A single Codex prompt can initiate the entire end‑to‑end pipeline. The agent performs these steps automatically:
- data annotation scanning and on‑the‑fly configuration patching (e.g., missing video FPS),
- secure model pull from Hugging Face using the provided token,
- zero‑shot baseline evaluation,
- generation of optimized LoRA training configs and launch of the TAO training container,
- post‑training evaluation and comparison to baseline.
In the demonstrated run the zero‑shot baseline accuracy was 54.41% (exact‑match, 4‑way). After one LoRA run (~30 minutes on eight NVIDIA A100 Tensor Core GPUs) the adapted model scored 87.14% — a +32.73 percentage point improvement — achieved hands‑free.
Optimizing with TAO AutoML
LoRA gives a strong initial gain, but final performance depends on hyperparameters such as learning rate, LoRA rank, dropout, batch size and schedulers. TAO AutoML runs structured sweeps with multiple search strategies (Bayesian optimization, Hyperband, combinations, and LLM‑guided suggestions). LLM‑guided searches can use an LLM brain (NVIDIA NIM, OpenAI, or compatible endpoints) and require an AUTOML_LLM_API_KEY or NVIDIA_API_KEY.
A follow‑up prompt instructed the agent to run an AutoML sweep. The agent generated candidate trials with multiple strategies, launched experiments, tracked hyperparameters and evaluations, and recorded the best configurations. The AutoML sweep in the experiment ran 43 parallel trials in full parallel across multiple A100 nodes, taking about 19.5 hours in the test environment, and achieved 93.35% validation accuracy (+38.94 percentage points vs. baseline).
Hardware and runtimes
- LoRA single epoch: ~30 minutes on a node of NVIDIA A100 (80 GB) GPUs.
- AutoML sweep in the demo: ~19.5 hours to process 43 parallel trials across multiple A100 nodes on Oracle Cloud Infrastructure.
- For comparison: a full‑parameter SFT run took 3 hours 34 minutes on NVIDIA H100 GPUs in this setup.
Qualitative improvements
Beyond numeric gains, post‑training improved model behavior on ambiguous, real‑world edge cases such as nuanced road formation and road type classification. The adapted model made more accurate, contextually appropriate decisions in the demonstrated examples.
Deploying a post‑trained LoRA adapter
The Cosmos 3 Reasoner NIM provides a fast path to a production, OpenAI‑compatible Reasoner endpoint. NIM microservices are prebuilt, optimized containers; you can pass the post‑trained LoRA adapter directory into the NIM environment during deployment to serve the adapted Reasoner.
Resources and events
NVIDIA points users to the NVIDIA/cosmos and NVIDIA‑TAO/tao‑skill‑bank GitHub repositories, Hugging Face for Cosmos 3 Nano and Super checkpoints, and video tutorials such as “How to Run NVIDIA Cosmos 3 Reasoner NIM for Video Reasoning.” NVIDIA also scheduled a Cosmos Labs livestream on July 16 and Cosmos Labs Office Hours on July 30 for live Q&A.
Conclusion
Combining Cosmos 3 Nano’s reasoning tower with TAO agent skills and AutoML enables rapid, automated domain specialization of vision‑language models. With minimal prompts, the coding agent automates dataset fixes, baseline benchmarking, LoRA adaptation and systematic hyperparameter tuning, turning a multiday engineering process into a single‑day, largely hands‑free workflow.
