Generative AI workloads increasingly exceed the compute and memory available on a single GPU. NVIDIA’s TensorRT multi‑device inference allows a single TensorRT network to execute across multiple GPUs using NCCL‑backed distributed collectives while retaining TensorRT inference optimizations. TensorRT 11.0 and later officially support this capability.
The NVIDIA Dynamo‑Triton (formerly NVIDIA Triton Inference Server) 26.07 release enables the multi‑device inference capability for the TensorRT backend. A Triton KIND_MODEL instance can now own multiple GPUs, create per‑rank TensorRT execution contexts, CUDA streams, and NCCL communicators, and launch the ranks together for each request. The application calls one named model via a gRPC endpoint instead of coordinating GPU ranks itself.
For organizations deploying generative AI, this bridges the gap between multi‑GPU acceleration and a consumable inference service. Teams can trade additional GPU resources for lower request latency, preserve application interfaces and workflows, package the engine as a versioned Triton model, and keep rank and communicator lifecycle code out of clients. For latency‑sensitive generative media workflows, shorter time to result reduces user wait time and speeds up review‑and‑refine cycles.
Example: Cosmos 3 Nano with Ulysses context parallelism
The integration was demonstrated using NVIDIA Cosmos 3 Nano video generation for a long‑sequence workload. Diffusers orchestrate prompts, latents, classifier‑free guidance (CFG), scheduling, VAE decode, and frame postprocessing. Dynamo‑Triton serves the 36‑layer denoising transformer; TensorRT multi‑device inference uses Ulysses context parallelism to distribute 44,160 video tokens across up to eight NVIDIA GPUs.
The distributed Ulysses graph is compiled into each TensorRT plan before deployment. The Dynamo‑Triton TensorRT backend loads the versioned plan, creates the multi‑rank execution state, and exposes one gRPC model endpoint; the client sends a transformer request to that endpoint and does not coordinate the participating GPU ranks.
Cosmos 3 Nano highlights the boundary: the transformer accounts for 93.4% of single‑GPU generation time, so it is the highest‑impact stage to accelerate. Each of 35 denoising steps requires one negative (unconditional) and one prompt‑conditioned prediction for CFG, so the Diffusers proxy makes two sequential Triton calls per step — 70 transformer RPCs per generation. Each request carries prepared tensors and returns noise_patches to the application workflow.
Activating a context‑parallel distributed TensorRT plan in Dynamo‑Triton
The distributed graph is compiled into each context‑parallel TensorRT plan; Dynamo‑Triton activates that plan instead of converting a single‑device engine into a distributed engine. The single‑device baseline uses a standard GPU model instance on GPU 0. The two, four, and eight GPU variants use KIND_MODEL, enable the TensorRT backend multi‑device path, and identify participating ranks.
Example excerpt from the generated CP8 config.pbtxt:
name: "cosmos3_cp8" backend: "tensorrt" max_batch_size: 0
instance_group [ { kind: KIND_MODEL count: 1 } ] parameters [ { key: "enable_multi_device" value: { string_value: "true" } }, { key: "multi_device_gpus" value: { string_value: "0,1,2,3,4,5,6,7" } } ]
Ulysses partitioning and distributed collectives
The fixed Cosmos 3 Nano profile in this example produces 44,160 video tokens. At context‑parallel size eight (CP8), each rank processes 5,520 video tokens outside attention; the shorter 2,992‑token text path remains replicated. Within each of the 36 transformer layers, Ulysses changes the partitioning axis around attention so that every rank processes the full video sequence for a nonoverlapping subset of heads.
The engine is exported from PyTorch and compiled with Torch‑TensorRT. Three local converters lower export‑carrier operations to the TensorRT public distributed‑collective layer: reduce‑scatter, all‑to‑all, and all‑gather. Each accepted context‑parallel plan contains two initial reduce‑scatters, three all‑to‑alls in each of the 36 transformer layers, and one final all‑gather — totaling two reduce‑scatters, 108 all‑to‑alls, and one all‑gather.
End‑to‑end generation latency benchmarks
All four variants ran on the same healthy eight‑GPU NVIDIA system. The single‑device baseline (SD) used one GPU; CP2, CP4, and CP8 used two, four, and eight ranks. Every run used 1280×720 output, 189 frames at 24 FPS, and 35 denoising steps.
Each result included one warm‑up followed by five measured full generations. Timing covered prompt work, the 70 Dynamo‑Triton calls, CFG and scheduler updates, VAE decode, and frame postprocessing. Model loading and mp4 encoding were excluded from timing.
Key averaged results:
- SD (1 GPU): E2E mean 156.595 s; transformer RPC mean 146.192 s; RPC share 93.4%.
- CP2 (2 GPUs): E2E mean 87.999 s (1.78× speedup vs SD); RPC mean 77.548 s (1.89×).
- CP4 (4 GPUs): E2E mean 53.093 s (2.95×); RPC mean 42.661 s (3.43×).
- CP8 (8 GPUs): E2E mean 34.183 s (4.58×); RPC mean 23.993 s (6.09×); RPC share 70.2%.
End‑to‑end latency decreased from 156.595 seconds on one GPU to 34.183 seconds on eight GPUs, while transformer RPC speedup reached 6.09×. As RPC portion falls (from 93.4% to 70.2%), time spent outside the measured RPC path (prompt processing, scheduler updates, VAE decode, postprocessing) becomes a larger fraction of the total; that outside time remained roughly 10.2–10.5 seconds across configurations.
Output validation
All variants used the same seed and generation profile. Validation sampled frames 0, 47, 94, 141, and 188, checked format and temporal variation, and compared each context‑parallel output with the single‑device result. CP2, CP4 and CP8 met configured thresholds: mean absolute error (MAE) ≤ 25 and peak signal‑to‑noise ratio (PSNR) ≥ 18 dB.
Measured values: CP2 MAE 12.759, PSNR 21.111 dB; CP8 MAE 16.316, PSNR 19.400 dB. Outputs are not pixel‑identical but show the same coherent action across the clip (for example, a robot arm cleaning a plate).
Practical considerations
For product teams, these results show a practical option when response time matters more than minimizing GPUs per request. A Cosmos 3 generation that previously took more than two and a half minutes can finish in about 34 seconds while the application continues to use a conventional model‑serving interface.
Teams still need to weigh resource‑for‑latency tradeoffs. This benchmark does not measure concurrent request throughput, cost per generated video, or total cost of ownership (TCO); teams should evaluate those metrics against their SLOs and economics.
To reproduce the results, download NVIDIA Dynamo‑Triton 26.07 from NGC and use the TensorRT, Torch‑TensorRT, Diffusers and Cosmos resources referenced by NVIDIA.
Related resources include the Dynamo‑Triton TensorRT backend multi‑device guide, TensorRT Multi‑Device documentation, and the Dynamo‑Triton Model Repository documentation.
