As large language model (LLM) inference increasingly handles sensitive data and proprietary context in personal, enterprise, and regulated environments, processing must occur inside trusted execution environments. NVIDIA Confidential Computing (CC) enables this by combining memory‑encrypted confidential virtual machines (CVMs), confidential GPUs, and encrypted NVIDIA NVLink, allowing production AI inference on trusted hardware.
This article reviews CC‑aware adaptations made by inference frameworks such as TensorRT LLM for Blackwell‑based NVIDIA B200 GPUs, and presents a controlled methodology platform engineers can use to measure CC overhead on their workloads.
Choosing a workload to reveal CC overhead
The visibility of CC overhead depends on workload characteristics. High request volume can amortize fixed encryption costs by overlapping stalls, making direct effects harder to observe. To surface CC costs, pick a workload with long input context, extended output generation, and low concurrency. Long context stresses prefill data movement, long generation amplifies per‑token CC costs during decode, and low concurrency reduces the opportunity to hide costs across concurrent requests.
The performance engineering team used the following workload configuration for the evaluation:
- Model: nvidia/DeepSeek-R1-0528-NVFP4
- Inference framework: TensorRT LLM, PyTorch backend
- I/O sequence length: 32K input / 1K output
- Concurrent requests: 1, 2, 4, 8, and 16
- Parallelism: TP=8, EP=1, PP=1
- KV cache: FP8
Measuring CC overhead with a controlled CC‑on vs CC‑off comparison
To isolate CC’s performance impact, run the identical workload under two conditions: confidential compute disabled (CC off) and confidential compute enabled (CC on), keeping model, hardware, framework version, sequence lengths, parallelism, and concurrency constant so that CC state is the only variable.
Metrics used:
- Output throughput retained: 100 × (CC on output tokens/s ÷ CC off output tokens/s)
- Latency overhead Time Per Output Token (TPOT): 100 × (CC on TPOT ÷ CC off TPOT − 1)
The hardware and software configuration used for both CC‑on and CC‑off runs (summary):
- Hardware: NVIDIA DGX B200 system (8 NVIDIA B200 GPUs)
- Platform: Intel TDX
- Host OS: Ubuntu 25.10, kernel 6.17.0-20-generic
- Guest OS: Ubuntu 24.04.4 LTS, kernel 6.8.0-124-generic
- Guest vCPUs: 256, Guest NUMA: 2 nodes
- NVIDIA driver: 595.71.05
- VBIOS: FW 1.4.x [97.10.64.00.0C]
- GPU power limit: 1,000 W
- CUDA: 13.2
- TensorRT LLM: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22
- NCCL: v2.30
- OpenSSL: 3.6.0
- Orchestration: Docker Container + NVIDIA Container Toolkit
Performance results
Across concurrency levels 1–16, CC on retained 96.1%–98.2% of CC off output‑token throughput. Mean TPOT with CC enabled was within 1.2%–4.3% of the CC‑off baseline (i.e., CC introduced 1.2%–4.3% TPOT overhead). The source report includes figures showing throughput and TPOT relative to the CC‑off baseline for concurrency 1–16.
Causes of CC overhead and mitigations
The NVIDIA Blackwell confidential computing architecture enforces hardware security paths that change some runtime assumptions (memory movement, timing, scheduling, and multi‑GPU communication). TensorRT LLM implements CC‑aware adaptations to reduce the resulting overhead:
- Adapting host‑to‑device data movement
- Issue: In the B200 CC configuration, host→device transfers go through a software encrypted bounce buffer because the GPU cannot directly access protected CVM memory. Pinned memory thus loses its usual asynchronous transfer benefit and some copies can block the calling thread.
- Mitigations: TensorRT LLM uses CC‑aware memory selection, choosing pageable memory for affected paths instead of always using pinned memory. For device→host transfers, repeated token and sampling‑data readbacks are moved to an asynchronous worker so protected copies don’t block the main scheduler during decode (see TensorRT LLM PR #11573).
- Stabilizing kernel autotuner timing
- Issue: The autotuner usually uses CUDA events to compare candidate tactics. In the tested CC configuration, CUDA‑event timestamps produced an unstable timing signal, which could lead the autotuner to pick a slower tactic.
- Mitigation: Under CC, TensorRT LLM uses the GPU %globaltimer for tactic measurements while retaining CUDA events outside CC (see TensorRT LLM PR #11657).
- Choosing CC‑aware multi‑GPU communication
- Issue: NVLS (NVLink SHARP) multicast is not available in B200 CC configurations. Without NVLS, NCCL_SYMMETRIC loses its multicast benefit and may still incur memory registration and cross‑rank synchronization costs before falling back to a non‑multicast collective path.
- Mitigation: Frameworks targeting CC should detect NVLS availability and choose communication algorithms that minimize latency for the message size, topology, and workload characteristics.
Recommendations for deployment
The evaluation shows confidential computing can be enabled with minimal performance loss if frameworks and the CC environment are optimized together. Practical guidance:
- Treat security configuration and inference optimization as a single, full‑stack engineering problem.
- Enable Confidential Computing, attest the environment, and run CC‑on vs CC‑off benchmarks using the exact workload you plan to serve.
- Consult NVIDIA Trusted Computing documentation and the latest TensorRT LLM release notes for CC guidance and feature updates.
Conclusion
With CC‑aware adjustments in data movement, autotuning, and multi‑GPU communication, confidential DeepSeek‑R1 inference preserved more than 96% of CC‑off output‑token throughput and kept per‑token latency overhead below 5% on eight NVIDIA B200 GPUs. As organizations move private inference into production, security settings and inference tuning should be coordinated as one deployment effort.
Acknowledgments
Thanks to Dan Hansen, Sheel Pethe, Samuel Mendoza‑Jonas, Moein Ghaniyoun, Vidhya Krishnan, Avinash Ahuja, Laikh Tewari, Laura Martinez, and Matheen Raza for engineering contributions, technical guidance, analysis, and review.
