A vehicle’s perception stack is shaped by the vehicle that carries it. Moving the same perception software to a different carline or variant changes how the system sees the world: sensor placement, calibration, fields of view, occlusions, body geometry, timing, and coverage all shift. As a result, features such as traffic lights, curbs, or pedestrians may fall into different parts of the image or become harder to observe. When an autonomous stack is expanded across carlines, developers must handle those differences even if the underlying perception code is unchanged.
Collecting and labeling a new real-world dataset for every carline is costly and often infeasible early in vehicle development: new fleets may not yet exist and rare conditions are hard to capture. Real-world driving data remain essential for grounding and validation, but synthetic data can help teams adapt models before a full target-carline dataset is available.
The carline-adaptation problem is therefore to prepare a perception stack for a vehicle that may not exist yet, and scale software across platform variants without collecting and labeling large datasets for each one.
How NuRec helps
NVIDIA Omniverse NuRec makes adaptation more practical by starting from recorded real-world drives, reconstructing each scene, and rendering new camera views for a target vehicle configuration. Using NuRec developers can reuse real drive data to answer questions such as:
- What would this scenario look like from the target rig?
- Which existing drives provide good coverage for the new vehicle?
- Where do geometry changes create weak spots?
- Which gaps require real data collection?
The workflow shown in NVIDIA’s tutorial follows four steps: pairing a reconstructed drive with a target rig configuration, rendering target views, refining frames with NVIDIA Harmonizer, and training a perception model on the output. The tutorial uses reconstructed scenes from the Physical AI NuRec Dataset and companion nurec-skills repository.
NuRec capabilities relevant to carline adaptation
NuRec reconstructs environments using 3D Gaussian splatting (gsplat) from sensor measurements and can render novel views. Two capabilities are especially useful:
- Novel-view synthesis: NuRec’s gsplat renderer projects Gaussians through a specified camera model, letting users change extrinsics, intrinsics, field of view, and lens model (pinhole, fisheye, f-theta, etc.).
- Reusable scene data and annotations: the reconstructed scene retains original rig trajectories, per-camera calibration, dynamic object tracks, and map data. These assets help align or adapt labels (objects, lanes, traffic lights, road boundaries) in newly rendered views.
Four-step workflow to adapt a perception model
Step 1 — Download a reconstructed scene
The Physical AI NuRec Dataset on Hugging Face contains over 1,500 neural-reconstructed driving scenes. Each scene is roughly 20 seconds long and was reconstructed from six source cameras: a 120° front-wide view, a 30° front-telephoto view, 120° cross-left and cross-right views, and 70° rear-left and rear-right views. The dataset is gated: accept the license, authenticate with a Hugging Face token, and download a scene.
The NVIDIA/nurec-skills repository contains helper skills for downloading Physical AI datasets, rendering with NuRec, and refining frames.
Step 2 — Render from the target carline’s sensor rig
A sample synthetic target rig is provided with the public sample; each sensor entry defines a colon-delimited camera name, resolution, an F-Theta lens model, intrinsic parameters, and a camera-to-rig pose. NuRec converts the colon-delimited name to a logical camera ID (for example camera:front:synthetic:120fov becomes camera_front_synthetic_120fov). To render additional cameras, add new sensor entries to rig.sensors with unique names and parameters.
The tutorial demonstrates exporting a custom rig trajectory from a USDZ scene and running a sparse render for one target camera to validate pose, field of view, and timestamps. After review, render the full sequence with frame-step=1 for each validated camera. The documentation notes that a target camera need not correspond one-to-one with any of the six source cameras: different position, orientation, resolution, or FOV is permitted, although cameras placed far outside the observed trajectory or toward poorly covered areas may produce lower-quality renders.
Step 3 — Refine rendered frames with NVIDIA Harmonizer
Neural rendering often produces view-dependent artifacts, inconsistent color or tone, and poorly reconstructed dynamic objects. NVIDIA Harmonizer is a public, temporally aware post-processing model that corrects such artifacts and improves visual consistency, but it does not fix incorrect calibration or recover scene areas that were never reconstructed.
The nurec-fixer skill in NVIDIA/nurec-skills covers setup, inference, evaluation, and optional Harmonizer fine-tuning. Harmonizer requires accepting model licenses, cloning its repository, building a runtime container, and downloading released checkpoints. Typical usage runs one camera sequence at a time and writes harmonized output beside the input frames.
Step 4 — Train and validate the perception model
After creating NuRec-rendered datasets for a new carline, validate them before incorporating them into perception training. Treat the synthetic dataset similarly to real camera data and prepare it according to the perception model’s input requirements.
NVIDIA internal evaluation
NVIDIA applied this workflow in an internal automated driving program that needed to support a new camera configuration before target-carline data was available. The team used an existing library of drives from a different vehicle and rendered target views with NuRec. After training on the NuRec-rendered synthetic data, they observed relative gains in object-detection precision and recall compared with a zero-shot baseline; the reported results included improvements for VRU (vulnerable road user) categories.
Automating the workflow with nurec-skills
The NVIDIA/nurec-skills repository packages the main steps as agent skills: locating and downloading Physical AI scenes, using NCore, running NuRec, and applying DiffusionHarmonizer. The nurec-carline-adaptation skill adds a compatibility and provenance layer but does not redistribute NuRec sources or models. A compatible coding agent can run validate-and-smoke-test sequences (export trajectory, run sparse camera render, stop before full render on failure).
Preparing source data for reconstruction
Reconstructing a captured drive requires synchronized video from the recording rig and corresponding metadata. The tutorial assumes the following layout: seven MP4s (front_wide.mp4, front_tele.mp4, cross_left.mp4, cross_right.mp4, rear_left.mp4, rear_right.mp4, rear.mp4) and three Parquet files (calibration_estimate.parquet with lens parameters and mounting positions; egomotion_estimate.parquet with vehicle poses; object_fused.parquet with moving objects and tracks). All three Parquet files are required by the companion converter and reconstruction recipe used in the tutorial; other workflows may accept different inputs.
The companion converter maps source files into NCore V4 components (CameraSensorComponent, IntrinsicsComponent, PosesComponent, MasksComponent, CuboidsComponent) and writes a separate-sensors profile that creates one camera archive per converted camera plus shared archives for poses, intrinsics, masks, and cuboids. The tutorial specifies coordinate conventions and timestamp handling (timestamps in integer microseconds, distances in meters, poses as valid SE(3) transforms).
Run the converter, validate the generated NCore manifest with the provided validator script, and inspect the sequence with the public NCore viewer.
Generate NuRec auxiliary data
The converted NCore sequence needs inferred semantic segmentation, metric depth, and ego masks for the camera-only reconstruction workflow. NVIDIA provides an nre-tools-ga NGC container with ncore-aux-data to produce these auxiliary products using mask2former for segmentation and depthanythingv2 for depth, and option flags to create per-camera ego-masks and metadata JSON. The outputs are zarr.itar archives such as <sequence-id>.aux.sseg.zarr.itar, <sequence-id>.aux.depth.zarr.itar, and <sequence-id>.aux.egomask.zarr.itar.
Getting started
- Download a USDZ scene and the sample target rig from the Physical AI NuRec Dataset.
- Render the scene through the sample rig, then replace the rig JSON with the target carline configuration and render again.
- Review NuRec documentation for setup requirements, supported hardware, validation, and rendering instructions.
- For reconstruction, follow Reconstruct an AV Scene and the NCore conversion and auxiliary-data guidance in NVIDIA/nurec-skills.
- Use NVIDIA Harmonizer for temporally consistent postprocessing.
Public software, containers, datasets, and model artifacts require applicable NVIDIA NGC or Hugging Face accounts and acceptance of their licenses. The NuRec runtime is available in the nre-ga container on NGC.
