Tools

AI-generated text

NVIDIA introduces Cosmos workflows for synthetic training data

NVIDIA Cosmos Data Factory combines five cloud-native blueprints with six reusable skills for preparing and generating training data for physical AI.

NVIDIA introduces Cosmos workflows for synthetic training data

Training AI systems that operate in the physical world requires diverse data, but recording rare defects, nighttime situations or dangerous events can be expensive and time-consuming. NVIDIA Cosmos Data Factory (CDF) aims to address that problem with workflows for processing data and generating synthetic examples.

CDF is part of the open NVIDIA Cosmos platform, which provides models, datasets, frameworks and skills for developing world models. Data Factory contains five cloud-native reference blueprints and six composable skills. Advanced AI agents can orchestrate them, allowing data to be produced from text instructions without continuous human steering once the system is set up, according to NVIDIA.

Setup and five use cases

All five blueprints rely on the CDF Orchestration skill, which coordinates workflow steps, deploys the necessary services and manages execution on the user’s computing cluster. In NVIDIA’s example, a coding agent first clones the required repository, checks the environment and performs setup. The preliminary check identifies missing requirements without changing the environment; a second instruction can then launch the selected data-generation task.

The five reference workflows address different training-data gaps:

  • Autonomous driving scenes: Autonomous Vehicle Dataset Generation searches for footage similar to existing driving videos, then creates variants with conditions such as rain, snow or fog. NVIDIA Cosmos Curator produces metadata, NVIDIA Cosmos Dataset Search retrieves relevant videos, NVIDIA Cosmos3-Super-Image2Video transforms them, and Cosmos Evaluator checks their physical plausibility.
  • Variations of images of people: Image Attribute Augmentation changes clothing types, colors or accessories. A large language model creates structured modification instructions, Cosmos Evaluator checks the requested attributes, and the system adds labels for person search and re-identification.
  • Video variations: Video Data Augmentation changes weather, lighting or surroundings while seeking to preserve the structure of a scene. Cosmos3-Nano generates a variant from a vision-language model’s description; Cosmos Evaluator checks for motion artifacts, the requested conditions and physical plausibility. Both source and generated videos receive detection, tracking and descriptive labels.
  • Rare-event videos: Event Video Generation can turn a single image into footage depicting events such as fires, falls or shoplifting. Cosmos3-Super-Image2Video preserves the viewpoint and layout; Cosmos Evaluator checks the event and motion and retries failed attempts. Passing clips receive automatic annotations.
  • Images of defective products: Defect Image Generation creates flawed examples from clean inspection images or three-dimensional USD assets. For 3D assets, NVIDIA Isaac Sim renders images and masks, while Qwen-Image-Edit creates clean reference images. Cosmos 3 Super uses these and the specified defect masks to generate flaws. The model comes fine-tuned for printed circuit boards, glass and metal; outputs also include manifests tracking their sources and generation settings.

Reusable skills and search

The workflows are built from six skills that can also be used independently: Auto Labeling annotates images and video; AnomalyGen creates defect images and masks from a handful of examples; Augmentation generates visual variations; Curation and Retrieval searches large collections; Simulation uses Isaac Sim and Omniverse Replicator to render labeled images from 3D scenes; and Orchestration runs and scales the services.

Cosmos Curator now supports multicamera video, inertial measurement unit and GPS data, with initial lidar support in its sensor library. NVIDIA plans to support additional lidar devices, radar and other sensors. Reusable Python components and Managed Ray on Slurm help teams coordinate processing and computing capacity.

Cosmos Dataset Search is open source under the Apache 2.0 license. It combines metadata, scene semantics, spatial relationships and temporal behavior using multi-embedding retrieval; adaptive agentic reranking is available in early access.

Costs and reported results

NVIDIA measured daily throughput on a cloud server equipped with eight NVIDIA RTX PRO 6000 Blackwell Series GPUs, at an estimated server cost of about $40 per hour or $960 per day. The company reported the following estimates:

  • The autonomous-driving workflow produced about 130 edge-case clips per day at $2.66 in GPU compute per clip. NVIDIA compared this with $125 per scenario collected on the road, making the generated clips about 45 times cheaper.
  • Image attribute augmentation produced about 42,649 annotated image pairs per day at $0.023 per pair, roughly 10 times cheaper than the $0.25 average for purchased third-party data.
  • Video augmentation produced about 305 annotated clips per day at $3.15 per roughly five-minute clip, compared with an average of $22 previously paid for third-party data.
  • Defect-image dataset assembly fell from two months to one day, a reported 60-fold speedup.
  • Event video generation produced about 384 annotated clips per day, at $2.50 and approximately four minutes per clip.

The figures cover compute costs only. Preparing source data, setup, agent use, storage and human review can add expenses.

NVIDIA also described results reported or targeted by organizations: at an offshore oil and gas site, recall of real-world anomalies improved by up to 24%, with an estimated 1,400-fold return on data-generation cost. For automated optical inspection program creation, organizations are targeting up to 85% less engineering effort and 40% fewer false calls. Another projection expects up to 90% fewer repeated physical test drives and up to 85% shorter AI regression test cycles. In one case, Cosmos Curator was associated with a time to market of under two weeks, compared with an estimated six months to build an equivalent pipeline in-house.

NVIDIA also announced a November 19 livestream on creating training-ready datasets. The planned event, featuring Voxel51 and Encord, is to cover Porsche’s use of Cosmos Data Factory through the Voxel51 platform and the collaboration between Encord and ANYbotics.