A TensorRT engine build can take from a few seconds to many minutes depending on model typing, deep tactic searches, and a cold timing cache on a new GPU SKU. Many integrations provide no progress reports and no way to abort early, which wastes GPU hours and leaves users or agent runtimes waiting on frozen terminals.
TensorRT exposes IProgressMonitor (available in NvInfer.h) to address this. The API lets the builder call back into a monitor during the build so applications can render live, nested progress and request cancellation. This article describes a minimal drop‑in implementation for Python and C++, adding a cancel path (Ctrl‑C or programmatic stop), and where to surface progress to terminals, IDEs, services, or agent runtimes.
All code examples here follow or are adapted from the two NVIDIA‑maintained samples:
- Python: samples/python/simple_progress_monitor/ (ResNet‑50, strongly typed network)
- C++: samples/sampleProgressMonitor/ (MNIST)
What IProgressMonitor provides
IProgressMonitor is an abstract base class the builder invokes during an engine build. You subclass it and override three methods (same semantics in Python and C++):
- phase_start / phaseStart(phaseName, parentPhase, nbSteps): reserve a progress row and record num_steps.
- step_complete / stepComplete(phaseName, step) -> bool: advance the bar; return False/false to cancel the build.
- phase_finish / phaseFinish(phaseName): tear down the row.
If parent_phase is non‑null the phase is nested, so the monitor observes a tree of progress instead of a flat list. The implementation must be thread‑safe because TensorRT may call the same monitor instance from multiple internal threads.
Attach the monitor to the builder by setting it on the IBuilderConfig in one call:
- Python: config.progress_monitor = MyMonitor()
- C++: config->setProgressMonitor(&myMonitor);
During a build the builder opens a top‑level phase (e.g., "Building Engine"), then nested phases (e.g., "Tactic Selection"). The builder repeatedly calls step_complete and inspects the Boolean return value from the monitor: true continues, false requests cancellation. On cancel the builder stops issuing new steps and unwinds active phases by calling phase_finish in reverse order.
What this tutorial builds
A minimal IProgressMonitor implementation in Python and C++, a cancel path via step_complete, and examples of routing progress updates to terminal, IDEs, HTTP services, or agent runtimes.
Prerequisites
- One NVIDIA GPU.
- TensorRT (current OSS release) and its Python bindings, or a build of the C++ samples.
- Python 3.10+ for the Python sample.
- TensorRT sample data: ResNet‑50 ONNX for the Python sample and MNIST ONNX for the C++ sample (provided in the sample‑data archive or under /usr/src/tensorrt/data in official NGC containers).
- A terminal that supports ANSI virtual‑terminal escapes (modern Linux shells, or Windows Terminal with VT enabled).
1) Subclass IProgressMonitor in Python
A minimal Python subclass tracks active phases and step counts and uses a Lock for thread safety. Crucially, Lock is required because TensorRT may call the monitor from multiple threads. Also note that phase_start cannot cancel a phase; the earliest cancellation point is the first step_complete for that phase. Only step_complete can return False to request cancellation.
The Python sample implements:
- phase_start: record phase state and render.
- step_complete: update current step, render, return not cancelled flag.
- phase_finish: remove phase and render.
2) Render nested progress bars with virtual‑terminal escapes
The renderer strategy used in the sample is terminal‑centric and works like this:
- Sort phases by nesting depth so children render under parents.
- Move the cursor up by the number of lines the PREVIOUS render printed and overwrite them in place. (Phases are added and removed, so previous line count matters.)
- For each phase print a line with a 40‑character progress bar, completed/total counts, and optional indent for child phases.
- Clear leftover lines when the render shrinks the tree.
The sample uses escapes such as \x1b[NA to move the cursor up N lines and \x1b[2K to clear a line. Do not redirect stdout to a file or pipe while using the terminal renderer; the escape codes will pollute logs. For non‑interactive sinks replace _render() with a structured emitter.
3) Add a cancel path
Cancellation is three lines once the monitor exists: install a SIGINT handler that flips the cancel flag, and have step_complete honor it. In Python the handler sets monitor._cancelled = True and prints a cancelling message. Wire the monitor into the builder config and call build_serialized_network(). On cancellation build_serialized_network() returns None; the builder unwinds at the next step boundary (which can take seconds in long tactic searches).
Applications should communicate the cancel latency to users (e.g., show "Cancelling…" while the builder unwinds). The same cancel flag can be set programmatically from an IDE Stop button, an agent timeout, or a CI cancel webhook.
4) The same pattern in C++
C++ mirrors the Python shape: a class deriving from nvinfer1::IProgressMonitor with a std::mutex for protecting an unordered_map of phases and an std::atomic<bool> cancelled_ flag. The methods phaseStart, stepComplete and phaseFinish behave as in Python, and requestCancel() sets the atomic cancellation flag. Attach it with config->setProgressMonitor(&monitor).
Using std::atomic matters because requestCancel() may be invoked from another thread or signal handler.
Where to wire it in real systems
IProgressMonitor is the single integration point between the builder and an application’s surfaces. The rendering and transport above the monitor are application‑specific; anything below it remains builder internals. Typical integrations:
- IDE extension: override _render() to emit LSP window/showProgress or $/progress notifications. Each phase becomes a progress token; step_complete maps to report messages; phase_finish to end.
- FastAPI / HTTP service: run builds in a background thread; have render push structured entries into an asyncio.Queue drained by the request handler over Server‑Sent Events. A cancel endpoint POST /builds/{id}/cancel calls monitor.requestCancel().
- Agent tool call: emit structured chunks per phase transition (e.g. {"phase":...,"step":...,"total":...}) into the tool stream so the agent runtime can render progress and call requestCancel() when budgets expire.
This pattern is especially important for agent runtimes: long builds must be observable and cancelable so agents can report progress, enforce time budgets, and stop cleanly.
Edge cases to handle
- Do not redirect stdout while the terminal renderer is active; escape codes will pollute logs. For non‑interactive sinks, use a structured emitter.
- phase_start cannot cancel; the earliest cancel point is the first step_complete of that phase. If cancellation occurs during a long phase_start, the builder may continue until the first step boundary.
- phase_finish may fire before all num_steps are reported — treat it as authoritative end‑of‑phase, do not assume current_step == num_steps.
- Cancel latency is bounded but not zero; the builder finishes the current step before checking the return value. Long tactic‑search steps can increase latency to seconds or tens of seconds.
- Thread safety is required; concurrent access to maps without synchronization will eventually crash or corrupt the display.
Get started
The fastest end‑to‑end way to try this:
git clone --depth 1 https://github.com/NVIDIA/TensorRT.git cd TensorRT/samples/python/simple_progress_monitor python3 simple_progress_monitor.py
This runs a live, animated ResNet‑50 engine build. Replace the sample's monitor class or attach a cancel handler as shown. The C++ equivalent lives in samples/sampleProgressMonitor/.
For production systems, replace the terminal renderer with your application's existing transport (LSP notifications, SSE, structured tool chunks). IProgressMonitor becomes the point where TensorRT build progress maps onto your application's progress model.
Learn more
- TensorRT Python API documentation for IProgressMonitor
- TensorRT C++ API documentation for nvinfer1::IProgressMonitor
- Python simple_progress_monitor sample
- C++ sampleProgressMonitor sample
- TensorRT GitHub releases and sample data
Using IProgressMonitor makes TensorRT builds observable and interruptible, which improves developer experience, agent behavior, and resource efficiency in long‑running build scenarios.
