NVIDIA DeepStream 9.1 introduces AutoMagicCalib (AMC) and Multi‑View 3D Tracking (MV3DT). Together they enable detections from multiple auto‑calibrated cameras to be projected into a shared 3D world coordinate system and maintain consistent object IDs across camera views. The release includes 13 modular agentic skills, reference applications, and sample datasets available on GitHub to accelerate multi‑camera video analytics development.
Why this matters
Single‑camera 2D tracking lacks reliable depth information and frequently loses objects once they leave the frame. Existing 3D approaches typically require manual calibration and complex computation. DeepStream 9.1 aims to simplify that workflow by automating camera calibration with AMC and by fusing multi‑camera observations with MV3DT to provide globally consistent tracking and unique IDs for every object in the monitored space.
Key additions in DeepStream 9.1
- 13 agentic skills to speed vision‑AI development and deployment.
- MV3DT skill: an end‑to‑end multi‑camera 3D tracking capability that provides consistent, cross‑view object tracking.
- AutoMagicCalib (AMC): an automated calibration skill that reduces manual effort and errors.
- NVIDIA JetPack 7.2 support for accelerated vision AI on Jetson edge platforms (for example Orin and Thor).
- Open‑source availability through a unified GitHub repository containing source code, skills and reference apps.
How MV3DT tracks objects across cameras
- Projection and fusion: each calibrated camera back‑projects 2D detections into 3D world coordinates using a 3×4 projection matrix and a ground‑plane assumption. The system fuses per‑camera 3D projections to ensure globally consistent tracking.
- Multi‑view association: cameras share tracklets over MQTT (Message Queuing Telemetry Transport). The multi‑view association algorithm matches tracklets by proximity in 3D world space and assigns a single global object ID when multiple cameras observe the same person.
- Outputs: On‑Screen Display (OSD) with tiled camera grid and 2D/3D overlays; Bird’s‑Eye View (BEV) top‑down map showing trajectories in world coordinates; Kafka messaging with per‑frame protobuf metadata (sensor ID, object ID, 3D bounding box, confidence) for downstream use.
- For algorithmic details on multi‑view association and 3D fusion, see the research paper “Fully Distributed Multi‑View 3D Tracking in Real‑Time” (Hernandez et al., 2026).
AutoMagicCalib (AMC): automating camera network calibration
MV3DT requires accurate 3×4 projection matrices for each camera. AMC automates calibration by analyzing tracked object trajectories from existing videos or streams instead of requiring physical calibration patterns in the space.
AMC pipeline stages:
- Per‑camera trajectory extraction: DeepStream detects and tracks objects; AMC collects those trajectories.
- Single‑view calibration and rectification: for each camera, AMC estimates intrinsic parameters (focal length, principal point, lens distortion) and produces a rectified view.
- Multi‑view tracklet matching: AMC matches tracklets across cameras using provided alignment points and the layout map as anchors.
- Bundle adjustment: jointly refines all camera parameters to minimize global reprojection error.
Optional VGGT refinement: AMC can optionally run the Visual Geometry Grounded Transformer (VGGT) model‑based calibration when object movement is limited or learned geometry yields more robust results. Users only need to supply a layout image and a few alignment points by selecting corresponding landmarks in camera views and on the map. AMC is provided as a microservice with REST APIs and a web UI.
DeepStream agent skills and deployment workflow
DeepStream 9.1 skills are built to work with coding agents (for example Claude Code, Codex or other agents). Instead of manually editing configs and running scripts, developers can issue natural‑language prompts and the agent performs setup, configuration, and execution.
MV3DT skill responsibilities include:
- Validating prerequisites (OS, GPU driver, Docker runtime)
- Pulling or building the DeepStream container
- Installing Kafka and Mosquitto
- Downloading detection model weights (PeopleNetTransformer, RT‑DETR, PeopleNet v2.6.3)
- Generating DeepStream pipeline configuration from the dataset
- Automatically triggering AMC if calibration files are missing
- Launching the full multi‑camera tracking pipeline
AMC skills include:
- amc‑setup‑calibration‑stack: pulls the AMC microservice container and starts the service stack (web UI + REST API)
- amc‑run‑sample‑calibration: runs end‑to‑end calibration on the bundled sample dataset
- amc‑run‑video‑calibration: runs calibration on a directory of video files plus layout image and alignment points
- amc‑run‑rtsp‑calibration: calibrates directly from RTSP streams
Quick start summary
Prerequisites: Ubuntu 24.04 (x86_64), NVIDIA driver >= 580, Docker with NVIDIA Container Toolkit, Claude Code or Codex installed and authenticated, NGC API key to pull DeepStream/AMC containers from nvcr.io, Hugging Face key to pull VGGT model (optional), optional X11 or VNC for display (headless mode supported).
Typical steps:
- Clone the DeepStream repo and install skills into your coding agent’s skill directory.
- Launch your coding agent (for example claude or codex) from the repo root and interact via natural language prompts.
- For the provided 12‑camera sample dataset (calibration included) the MV3DT skill builds the environment, downloads models, starts Kafka and Mosquitto, launches the DeepStream container and begins tracking. A tiled 12‑camera OSD and a BEV trajectory map are shown if display is available; otherwise output videos are saved.
- For custom video datasets, provide a folder of time‑synchronized camera files named sequentially and a layout.png; the agent will detect missing camInfo/*.yml files, invoke AMC to run calibration (including a manual alignment step via the AMC web UI), and then run MV3DT using the generated calibration files.
Outputs and downstream use
A successful MV3DT deployment yields:
- Live OSD with tiled camera feeds, 2D/3D overlays and consistent object IDs
- Live BEV top‑down trajectory map with screenshot/recording options
- Kafka metadata stream on the mv3dt topic with per‑frame protobuf messages (object IDs, 3D bounding boxes, confidences, sensor IDs) suitable for analytics, dashboards or alerting
- Saved videos (tiled OSD and BEV trajectory) when file output is enabled
Availability and resources
All source code, skills, reference applications and sample datasets for MV3DT and AMC are available in the NVIDIA DeepStream GitHub repository as part of the DeepStream 9.1 release. The MV3DT reference app is at src/apps/reference_apps/deepstream-tracker-3d-multi-view and skill implementations are in skills/ (including skills/deepstream-run-mv3dt and the amc‑* skills).
For community support and discussion, consult the NVIDIA DeepStream Developer Forum.
