NVIDIA TensorRT Model Connect is an open collection of reference implementations intended to simplify running supported open models with NVIDIA TensorRT inside native C++ applications. The project aims to remove the need to rebuild model-specific conversion, preprocessing, post-processing, and runtime code for every model: developers can use, inspect, modify, and extend the reference implementations.
Model Connect is not a new inference framework and does not replace TensorRT; rather, it serves as a bridge between end-to-end model workflows and TensorRT’s GPU-accelerated execution engines.
Two-command deployment flow
Model Connect separates deployment into two phases and uses a single bundle artifact between them:
- Build the bundle (Python CLI)
-
For a supported model, the first phase builds a deployment bundle from a Hugging Face model ID or a local checkpoint. Example command:
trtmc build Qwen/Qwen3-0.6B -o qwen3-0.6B.bundle
-
The produced bundle contains the TensorRT engines and model-specific assets required at runtime.
- Load and run (C++)
-
In the second phase a native C++ application loads the bundle and works with task-level inputs and outputs. Example C++ snippet:
#include <trtmc/pipeline.h> auto pipeline = trtmc::load("qwen3-0.6b.bundle"); auto result = pipeline->generate("Explain why native inference matters.", {.max_new_tokens = 20}); std::cout << result.text << std::endl;
-
Model Connect handles checkpoint mapping, TensorRT engine construction, preprocessing, runtime orchestration, and post-processing. The deployed application runs natively and does not require PyTorch or a Python interpreter in production.
Two API levels
Model Connect provides two C++ API levels:
-
Semantic API (task-level): accepts familiar inputs and outputs such as prompts, images, and audio while Model Connect performs model-specific preprocessing, execution, and post-processing. This is a simple starting point for most applications.
-
Module-level API: exposes named tensors and individual TensorRT components so developers can customize the inference pipeline with fine-grained control. Both APIs share the same underlying implementations, allowing a developer to begin with the simpler task-level interface and only dive deeper when necessary.
Extending with custom GPU kernels (TVM FFI)
TVM FFI provides a language-agnostic interface for invoking GPU kernels without tightly coupling the caller to the kernel’s implementation framework. When used through TensorRT Model Connect, TVM FFI lets you replace a targeted portion of a model with a custom GPU kernel while TensorRT executes the remainder of the pipeline. This simplifies integrating specialized or newly developed kernels without rebuilding the entire application around a separate runtime. The project’s “Bring Your Own Kernel” tutorial shows a worked example.
Reference implementations for the open model ecosystem
Each model implementation in Model Connect serves three purposes:
- run a supported open model in a native TensorRT-enabled application;
- provide a complete, inspectable implementation of the model and its inference pipeline for learning and reuse;
- enable extension for related architectures, custom checkpoints, or specific application requirements.
This approach gives the wider community a clearer path from a model ID to TensorRT deployment: application developers can start from working code, and contributors can reuse patterns to add support for new models instead of starting from scratch.
Built AI-native to keep pace with fast change
The open model ecosystem evolves rapidly, and the project is built as an AI-native software effort: coding agents generate implementation code, tests, integrations, and documentation under human direction and review. This enables parallel development and validation of multiple model implementations while preserving consistent architecture and user experience.
Model Connect uses nightly releases to shorten the time from a new model or community contribution to an available implementation; automated validation gates the releases.
Full TensorRT workflow and performance
Because Model Connect is built on TensorRT, performance is a core consideration. For supported and validated workloads, Model Connect can deliver faster inference than torch.compile, and each implementation is continuously tested and optimized as the project evolves.
At the same time, usability is emphasized: the workflow covers finding a model ID, building the model bundle, loading it from C++, and adapting it as needed, providing an accessible route to high-performance TensorRT inference while keeping the pipeline inspectable and customizable.
Getting started
The GitHub repository NVIDIA/TensorRT-Model-Connect contains supported implementations and instructions for building a model bundle. You can use an implementation as-is, adapt it to your application, or contribute new model support.
The project also suggests an AI-native quick-start flow that automates cloning the repository, detecting GPU compute capability, adjusting the development Docker image, building and running the container, installing TensorRT Model Connect, compiling the CLI and native backends for the detected SM, and running an end-to-end Qwen/Qwen3-0.6B smoke test to produce a report.
Conclusion
TensorRT Model Connect provides reference code and a two-phase deployment path to simplify bringing open models into native C++ applications with TensorRT. It offers two API levels, support for custom kernels via TVM FFI, AI-assisted development practices, and nightly releases to respond quickly to changes in the open model ecosystem, aiming for a consistent deployment path wherever TensorRT is available.
