NVIDIA describes how video-based AI can be taken beyond observation and summarization to initiate coordinated, automated actions grounded in organizational knowledge. The approach composes the Video Search and Summarization (VSS) Blueprint and the Retrieval-Augmented Generation (RAG) Blueprint with the NemoClaw agent framework so that enterprises can move from "What does the video show?" to "What should we do about it, and how do we coordinate that action at scale?"
Components and their roles
- NVIDIA NemoClaw: a collection of open blueprints for building autonomous agents. It enables domain-specialized, always-on agents that are faster, safer, and more cost-efficient across digital and physical workflows.
- NVIDIA Blueprints: customizable reference workflows for enterprise-scale, agentic AI pipelines, combining specialized microservices, optimized models, and composable APIs.
Key Blueprints used in the workflow:
- NVIDIA Metropolis Blueprint for Video Search and Summarization (VSS): ingests streaming or archival video, generates captions and visual metadata, and supports semantic search, interactive Q&A, and event summarization.
- NVIDIA AI Blueprint for Retrieval-Augmented Generation (RAG): indexes proprietary documents (manuals, policies, SOPs, reference data) into a GPU-accelerated vector store for fast semantic search.
How VSS captures intent, retrieves knowledge, and generates reports
VSS provides guided, context-aware video analysis through agent tools. The process begins with human-in-the-loop (HITL) prompts that capture user intent: what to analyze, which events matter, and which objects to track. The agent then queries the RAG Blueprint for organizational context and produces a structured, timestamped report.
Three agent tools operate together:
- Long video summary (LVS) video understanding tool: performs hierarchical summarization of long videos with mandatory HITL parameter collection (scenario, events of interest, objects to follow, optional knowledge-retrieval query).
- Knowledge retrieval (frag) tool: calls the RAG Blueprint to fetch organization-specific context from documents, policies, and reference data; embedding, reranking, and vector search are handled within the RAG.
- Report generation tool: combines video analysis with retrieved context into a structured report with timestamps, narrative analysis, citations, and optional HITL confirmation or edits before finalization.
These tools collect intent via HITL, query the RAG for context, process the video with that context, and hand the results to the report generator.
From video findings to assessments and recommended actions
To illustrate, the article describes a "healthy eating coach" example that analyzes meal-preparation videos, assesses eating habits, and returns concrete, tracked next steps.
Process summary:
- A user uploads a meal video to the VSS interface.
- NemoClaw reads the vss-generate-video-report-rag skill definition and hands the request to the VSS agent. The agent guides the user through HITL prompts in the terminal to collect parameters: what to analyze, scenario, events, objects to track, and an optional RAG query (e.g., nutritional guidelines).
- With parameters confirmed, the LVS tool queries the RAG Blueprint for relevant reference documents. The RAG returns matching materials via its vector search and reranking.
- LVS summarizes the video hierarchically and embeds the reference knowledge into its findings. The report generation tool produces a structured, timestamped report containing detected events, narrative analysis grounded in reference material, citations, and concrete recommended actions.
NemoClaw then reads the finished report and turns it into coordinated action. It presents the completed analysis with links to Markdown and PDF reports and the video playback, and it can automatically create a Jira ticket that summarizes the findings and recommended adjustments with an appropriate priority and assignment so actions are tracked to completion.
This downstream action generalizes beyond Jira: NemoClaw can create tickets with correct priority and assignment, escalate or summarize patterns across multiple runs, bundle supporting evidence for review or compliance, and route gaps to appropriate follow-up workflows. The report becomes a trigger for coordinated action across existing systems.
Architecture (four layers)
- Orchestration: NemoClaw agent, the vss-generate-video-report-rag skill, and HITL prompts.
- VSS agent: video I/O, search, understanding, LVS, knowledge retrieval, and report generation. Knowledge retrieval is integrated into the agent.
- RAG Blueprint: NVIDIA RAG API, Milvus vector database, NVIDIA Nemotron reranking NIM, and indexed reference and organizational documents.
- LLM fusion: enrichment of the VSS summary with context retrieved through the RAG Blueprint.
Data flows downward through the system: the agent's tools orchestrate calls to the LVS service and the RAG Blueprint, both feeding into the report generation tool for final output.
How to deploy the VSS agent with knowledge retrieval — prerequisites and steps
Prerequisites:
- NVIDIA GPU(s) with at least 24 GB VRAM
- Docker Engine and Docker Compose v2
- NGC API key (ngc.nvidia.com)
- NVIDIA Build API key (build.nvidia.com)
- A deployed, reachable RAG Blueprint and the collection name
- NemoClaw installed for programmatic access
High-level steps summarized from the post:
- Clone the VSS repository and authenticate with NGC.
- Configure the LVS profile .env file. Set VSS_AGENT_CONFIG_FILE to config_rag.yml to enable the frag knowledge-retrieval tool, and provide RAG_SERVER_URL, RAG_API_KEY, and KNOWLEDGE_COLLECTION values.
- Deploy the VSS stack with docker compose (the compose profile is selected from the .env file) and ensure required data directories are created and writable.
- Verify service health (VSS agent, LVS backend, RT-VLM, LLM NIM). Note that the NIM may take 5–15 minutes to load.
NemoClaw setup:
- Run the NemoClaw installer from the repo root with NEMOCLAW_PROVIDER specified (use build for NVIDIA-hosted models). This command onboards NemoClaw, configures the model provider, applies the VSS sandbox policy (allowing sandbox access to the VSS agent), installs repo skills including vss-generate-video-report-rag as an OpenClaw plugin, and prints the OpenClaw UI URL.
- To use a custom OpenAI-compatible endpoint (for example a local vLLM), set NEMOCLAW_PROVIDER=custom and provide NEMOCLAW_ENDPOINT_URL and COMPATIBLE_API_KEY.
- Test the end-to-end workflow by connecting to the NemoClaw sandbox and using the OpenClaw UI: start a new session (/new) and submit a request such as "I want to generate a video summary report for <VIDEO_NAME>". The agent will collect HITL parameters, run LVS and RAG, and optionally create Jira tickets or send notifications based on results.
Latency and performance
Adding NemoClaw orchestration and HITL parameter collection introduces minimal latency. The HITL phase is asynchronous, so once parameters are confirmed, video processing proceeds without additional blocking overhead from the orchestration layer.
Industry use cases and partner deployments
- Computacenter deployed the full DETECT → REASON → ACT pipeline on a Run:AI cluster for predictive maintenance. They used VSS 3 to analyze drone, borescope, and thermal inspection footage, RAG Blueprint for OEM context, and NemoClaw to auto-draft Maximo work orders—reducing footage-to-work-order time from 30–45 minutes to roughly 19 seconds across four asset classes.
- VAST Data uses NemoClaw to orchestrate a real-time VSS pipeline on the VAST DataEngine, processing live game streams with VAST RAG over VastDB and vectors; the solution is built on the NVIDIA blueprint with cosmos-reason2 and Nemotron and runs end-to-end on NVIDIA DSX AIR.
Benefits of actionable video AI
Combining VSS 3, the RAG Blueprint, and NemoClaw changes how enterprises can use video analytics. Instead of producing static reports that require manual interpretation and manual follow-up, the integrated system treats video understanding as the entry point to coordinated workflows that:
- Speed response times—from hours or days to minutes or seconds.
- Scale to analyze thousands of videos across multiple sources and surface enterprise-wide patterns.
- Ensure consistency by applying reference knowledge (policies, procedures, regulations) uniformly.
- Provide accountability by tracing decisions back to video evidence and source documents.
- Automate routine workflows—alerts, ticket creation, and documentation—without manual intervention.
Getting started
The post recommends cloning the VSS repo, deploying the VSS agent with knowledge retrieval enabled, pointing the RAG Blueprint to your documents, and connecting NemoClaw to enterprise systems such as Jira, Slack, a database, or a ticket queue. Begin with a single recurring loop from video to decision to action, pilot it end-to-end, and expand from there.
The key architectural principle is composition: specialized systems (VSS and RAG blueprints) connected through clean APIs and orchestrated by NemoClaw deliver capabilities that exceed what any single model or service can provide, while preserving modularity, scalability, and governance required for enterprise deployments.
