The Global Performance Tuner is an automatic, accuracy-aware, end-to-end performance tuning feature introduced in NVIDIA TensorRT 11.1. Its purpose is to represent alternative scheduling strategies as explicit, searchable build routes and to select the fastest route that still meets a user-defined accuracy requirement in the target deployment environment.
Core concepts: knobs and build routes
The central idea is to expose meaningful compiler configuration points (knobs) that change parts of the compilation pipeline—heuristics, fusion decisions, or codegen paths. Each tunable knob has a finite set of allowed values and a default. A build route is a concrete combination of knob values identifying a requested path through the tunable parts of the builder.
A knob combination is a specification of a build route, not a complete description of the final engine: TensorRT still executes the remaining builder steps (for example, tactic selection) and may reconcile knob values with network constraints.
A simple illustrative example: a compiler with passes f1, f2, f3 and a binary knob p controlling f2’s fusion strategy (p=1 horizontal fusion, p=0 vertical fusion) yields two build routes. Different models, tensor shapes, or GPUs can favor different routes; therefore both candidate engines must be built and measured to determine which is better for the target case.
APIs and the trtexec sample
TensorRT exposes APIs to apply/inspect a single build route and to discover the tunable configuration space. The relevant C++ APIs are:
- bool IBuilderConfig::setBuildRoute(char const* buildRoute);
- char const* IBuilderConfig::getBuildRoute() const;
- char const* IBuilderConfig::getAllBuildRoutes() const;
These APIs enable custom tuners; NVIDIA provides a sample implementation in trtexec.
High-level trtexec workflow:
- Given an input model and a tunable build-route expression (regex-like), trtexec expands the expression into concrete build routes according to the search algorithm.
- It iterates over the fixed routes: for each route, the model is built, and accuracy and performance are measured.
- Routes that crash during build or fail accuracy checks are excluded; the fastest measured, eligible engine is selected.
A Python implementation will be available in Torch-TensorRT 2.15.
Search space size and empirical gains
TensorRT 11.1 exposes more than 200 knobs for tuning. Because combinations can grow combinatorially, practical search strategies are needed. In experiments reported by NVIDIA, across ten evaluated workloads the best eligible route found during tuning improved end-to-end performance between 1.14× and 1.63× compared with the default build route (default normalized to 1.0× per workload).
These results illustrate that different models reveal different optimization opportunities, and a single default scheduling strategy cannot be expected to be optimal universally. The Global Performance Tuner explores alternatives, filters by accuracy, and selects the fastest eligible engine measured in the deployment environment.
Accuracy–performance entanglement: a thought experiment
The tuner is accuracy-aware and uses an accuracy tolerance ε. Consider a model with four binary knobs controlling whether to insert Q/DQ (quantization) nodes at four locations, producing 2^4 = 16 candidate solutions. The fully quantized candidate may give the best performance but fail accuracy; the unquantized candidate gives best accuracy but worst performance. Depending on ε, the set of eligible candidates changes:
- ε = 0: only bit-wise identical outputs to the reference are accepted; possibly only the unquantized solution remains.
- ε = +∞: no effective accuracy restriction; all candidates remain eligible and performance-only optimization could select meaningless solutions.
- ε decreasing from +∞ to 0: the eligible set shrinks gradually; at each ε the tuner picks the fastest measured solution among the eligible candidates.
Thus the Global Performance Tuner does not remove the trade-off, but separates roles: accuracy defines which candidates are acceptable and performance ranks those acceptable candidates. Developers supply the accuracy boundary (metric, dataset, threshold), and tuning finds the best-performing evaluated build route within that boundary.
Best practices for effective tuning
Because the search space can be large, the following practices help keep tuning controlled, reproducible, and aligned with the target application:
- Choose search strategy by search-space size: fast scan evaluates the default route and varies one knob at a time; mixed starts with fast and then explores promising knob combinations; full evaluates the full Cartesian product and is suitable for small knob sets.
- Use --dryRun (supported for fast and full) to list concrete routes without building engines. dry run is not supported for mixed search because the second phase depends on measured data.
- Treat tuning as incremental and resumable: specify --tuningCacheFile so each completed iteration is recorded (route, GPU time, accuracy). Use --continue to resume an interrupted sweep.
- For debugging, --saveAllEngines preserves every engine produced, but it consumes substantial storage and is mainly for investigating failures or regressions.
- Tailor inputs and reference outputs to the deployment conditions that matter. Validate with representative inputs, boundary cases, and an accuracy metric that reflects real application quality requirements.
- Use the trtexec sample as a ready-to-use tuner, but consider building application-specific search policies: custom heuristics, filters, or pre-validation before comparing performance.
- Treat previous tuning results as guidance, not universal solutions: the best route depends on model, input shapes, target GPU, software stack and TensorRT/tuner versions. Re-tune after relevant changes.
Getting started and the broader impact
The Global Performance Tuner shifts TensorRT from a single universal scheduling strategy toward model-specific, evidence-driven optimization. By separating candidate generation (knobs) from route search, it helps avoid heuristic sprawl and makes optimizations measurable and reproducible under an explicit accuracy constraint.
To begin, consult the Global Performance Tuner documentation and examine the open-source trtexec implementation. Extend the samples with your own accuracy validation, search strategies or heuristics, or implement a custom tuner using the build-route APIs to suit your application’s accuracy and performance goals.
