AI-generated text

Agent-assisted development of a real-time endoscopic tool segmentation app on NVIDIA Holoscan

NVIDIA demonstrates an engineer-guided workflow where an AI coding agent builds and optimizes a real-time endoscopic tool segmentation application using Holoscan and HoloHub resources.

AI Newsroom · 19 August 2026 · 5 min read

NVIDIA describes an engineer-guided workflow in which an AI coding agent builds and optimizes a real-time endoscopic tool segmentation application on the Holoscan platform. HoloHub examples, documentation, and the Holoscan CLI provided a shared execution surface so that both the agent and the developer could discover and run the same commands.

Goal and starting point

The project goal was an end-to-end endoscopic tool segmentation application: live inference, visualization of segmentation masks, and statistical rendering. The work reused the MONAI endoscopic tool segmentation model and a Holoscan sample video, and the team verified that the existing monai_endoscopic_tool_seg application ran locally. The agent environment included:

Holoscan CLI with Bash execution permission (./holohub wrapper)
HoloHub repository with progressive disclosure documentation (agents.md) and examples
HoloHub development skills such as holohub-app-lifecycle and holohub-debug-build-run

Method: iterative, verifiable steps

The overall objective was decomposed into smaller, reviewable engineering iterations: confirming the environment, running model and video end-to-end, validating visualization, enabling repeatable latency measurement, and improving rendering throughput without regressions.

Iteration 1 — Minimal working application

The first prompt constrained reuse of model and data while defining the expected outcome: create a new Python HoloHub application that displays endoscopic tool tracking using the MONAI endoscopic tool segmentation model and sample data, show model-derived masks, coverage and timelines in a polished HoloViz overlay, and do not change model weights.

The agent inspected relevant HoloHub examples (notably monai_endoscopic_tool_seg, endoscopy_tool_tracking, surgical_scene_recon), generated the standard scaffold with ./holohub create, implemented the application graph, execution modes, tests and documentation using existing Holoscan operators, and built and ran the app via ./holohub run. The resulting application connected video replay, preprocessing, TensorRT inference, the SDK segmentation postprocessor, telemetry, and HoloViz; inference and mask postprocessing ran on every replayed frame and the overlay reported frame-derived measurements.

Agent processing time for this iteration was approximately 40 minutes. The developer reviewed the live app with the same CLI the agent used: ./holohub run endoscopy_tool_segmentation_dashboard visual --language python After code and visual review it was clear the overlay needed clearer measurements, prompting the next iteration.

Iteration 2 — Repeatable benchmarking

The second prompt turned the visual demo into a repeatable artifact by adding more meaningful statistics (tool area, mask motion, temporal intersection-over-union, edge entropy, FPS, bounding box position), removing static values, and adding a benchmark mode to record actual latency and plot results in Python. Figures should be exported to the build folder and shown interactively when supported.

The agent revised dynamic measurements and screenshot readability, and implemented explicit application modes. The app exposes three named modes discoverable via Holoscan CLI:

visual: run the full sample interactively at source pace
smoke: quick 60-frame headless recording with a finite verdict
benchmark: process 300 frames offscreen and export measurements and plots

Benchmark mode used Holoscan Data Flow Tracking for the path from the video replayer through preprocessing, inference, telemetry, offscreen HoloViz, and the rendered-frame sink, reusing ideas from the existing holoscan flow benchmarking module. Agent processing time for this iteration was about 20 minutes.

Iteration 3 — Investigate and reduce latency

The third prompt asked whether inference ran on every frame and requested latency reduction strategies, including reusing neighboring frame segmentations. The agent confirmed inference still ran every frame; although reusing masks could save inference work, it would require a policy for when stale outputs are acceptable. For this iteration the agent instead removed lower-risk dashboard overhead:

reused HoloViz input specifications and static coordinate tensors, while refreshing text and dynamic geometry per frame
queued the current 10-value GPU-to-host telemetry copy asynchronously in two pinned buffers and rendered using previously completed values

Comparing the baseline and optimized implementations on the same test system, the optimized version was faster in all five trials. Measured changes:

Rendered throughput: 204.0 FPS -> 306.9 FPS (50.5% higher)
Mean application-path latency: 4.891 ms -> 3.247 ms (33.6% lower)
P95 application-path latency: 6.273 ms -> 4.554 ms (27.4% lower)

Agent processing time for this iteration was around 30 minutes.

Final handoff and verification

After three iterations the developer asked the agent to commit the implementation and retain benchmark logs. The team reran the application and tests, verified output figures and headless results, and preserved the commit hash and dependency versions using ./holohub env-check and ./holohub env-info.

Ablation study: which resources matter?

The authors compared the same coding agent and sandbox across three scaffolding configurations. The ablation was last conducted on Aug. 1 using Codex 0.146.0 with GPT-5.6 Sol at max reasoning effort. Results for creating the first application iteration:

CLI + skills + docs/examples: agentic processing time 40 min, token usage 11M — outcome: separate application using standard Holoscan operators
CLI + docs/examples (no skills): processing time 65 min, token usage 20M — outcome: complete app but less efficient; agents often used generic Bash tools and required more trial-and-error
docs/examples (no agents.md, no skills): processing time 40 min, token usage 15M — outcome: app required rework; 3rd-party model config embedded incorrectly, Dockerfile didn’t leverage HoloHub base image, Holoscan optimized operators were ignored, resulting in ~2.6x slower performance

Overall, the combination of CLI, skills, and documentation/examples provided the best developer experience with the lowest resource overhead.

Conclusions

The presented co-development loop produced an engineering prototype built on an existing model, sample video, and Holoscan components. It runs end-to-end, offers multiple application modes and automated tests, preserves model weights, and records reproducible benchmark evidence. The core workflow pairs a shared CLI (./holohub), project-specific skills, and examples/documentation so that an agent and an engineer can iterate: the agent executes defined sequences and checks, while the developer sets goals, constraints and evaluates the evidence.