Concerns over data privacy, sovereignty and protecting models while they are running can slow enterprise AI adoption. NVIDIA Confidential Computing (CC) provides hardware-embedded and system-level safeguards to protect data, model weights and code during active inference while keeping performance close to non‑confidential execution.
What CC protects
CC is designed to maintain confidentiality and integrity for data, model parameters and runtime code during inference. The solution spans from silicon to interconnects and system software, delivering an end-to-end security layer for production AI workloads.
Hardware root of trust
NVIDIA Blackwell GPUs — including the NVIDIA RTX PRO 6000, HGX B200 and HGX B300 — are built with CC capabilities in hardware. Each GPU contains a private signing key fused during manufacturing and never exposed to software, firmware or the host, forming the basis of the attestation chain. The HGX B200 and HGX B300 configurations support confidential computing across multiple GPUs (up to eight) using encrypted NVLink.
Attestation before secret release
Before a confidential workload receives secrets (for example, a model decryption key), it must pass remote attestation. The NVIDIA Remote Attestation Service (NRAS) validates a signed evidence bundle — the GPU hardware report combined with CPU Trusted Execution Environment measurements (AMD SEV‑SNP or Intel TDX) — against a known-good reference integrity manifest (RIM). The attestation handshake is typically a one-time startup event; once the Confidential VM (CVM) is verified and running, attestation does not add per-request latency.
How CC affects inference performance
Two main factors can influence AI inference performance in CC mode on Blackwell GPUs:
- Secure work submission latency: Encryption and kernel launch overheads make smaller work units proportionally more expensive. Increasing the amount of work per GPU launch reduces this overhead's impact.
- Reduced host-to-device bandwidth: Workloads that depend heavily on transferring inputs to the GPU may be limited by the encrypted transfer bandwidth available in CC mode.
NVIDIA and upstream projects introduced several optimizations to mitigate these effects:
- CC-safe autotuner timing (FlashInfer): uses the GPU global timer register in CC mode so autotuners can accurately compare kernel candidates.
- Async device-to-host copy worker: SGLang moves per-step token readback off the scheduler’s critical path to restore compute/copy overlap that CC can otherwise serialize.
- Piecewise CUDA graph support: SGLang adds CUDA graph replay for prefill and mixed batches to reduce amplified kernel launch overhead in CC mode.
NVIDIA continues collaborating with inference framework communities to optimize their frameworks for CC performance.
Benchmark setup
The team measured CC performance using the Qwen 3.5 397B‑A17B model at FP8 precision on HGX B300 (Blackwell Ultra). The environment was a virtual machine with GPU passthrough. The baseline was Confidential Computing off; the experiment enabled Confidential Computing with all other variables held constant.
Key software and hardware details:
- Platform: Intel TDX
- Host OS: Ubuntu 25.10 (kernel 6.17.0-20)
- Guest OS: Ubuntu 24.04.4 LTS (kernel 6.8.0-124), 256 vCPUs, 2 NUMA nodes
- NVIDIA driver: 595.71.05; CUDA 13.2
- SGLang: docker.io/lmsysorg/sglang:v0.5.12-cu130 (PRs: 28251 SGLang and 3638 FlashInfer)
- NCCL v2.28.9-1; OpenSSL 3.6.0
- Orchestration: Docker + NVIDIA Container Toolkit
- GPU power limit: 1100W
Workload parameters covered realistic enterprise inference conditions:
- Input/output token lengths: 8192/1024 and 1024/1024
- Batch sizes: 4, 8, 16, 32, 64, 128 and 256 concurrent requests
- Inference framework: SGLang (Server)
Collected metrics included output throughput per GPU (tokens/sec), median Time To First Token (TTFT) in ms, and median Time Per Output Token (TPOT) in ms.
Benchmark results: small, single-digit overheads
Across all tested workload configurations, enabling CC produced minimal throughput and per-token latency overheads during steady-state inference. Relative performance changes (Δ% vs CC off) observed on HGX B300 with Qwen/Qwen3.5-397B-A17B-FP8 generally fell into single-digit percentage declines. For example, across concurrency levels from 4 to 256 and ISL/OSL settings of 1024/1024 and 8192/1024, throughput and TPOT deltas were typically between -1% and -8% depending on the configuration.
Summarized observations:
- Throughput/GPU reductions were frequently in the 2–6% range at small-to-medium concurrency levels.
- Median TPOT (steady-state per-token latency) showed similar single-digit degradations.
These results indicate that the security benefits of CC (encrypted communication, attested execution and hardware key protection) can be achieved with only modest impact on production inference performance.
Conclusion and next steps
Hardware-level Confidential Computing provides a robust foundation for protecting sensitive AI workloads without materially sacrificing inference performance. In the evaluated Qwen 3.5 + SGLang setup on Blackwell Ultra, CC yielded near-native performance across concurrency levels and sequence lengths, suggesting organizations can meet regulatory and data-protection requirements while maintaining efficient production inference.
For more technical details and deployment resources, NVIDIA points to its Confidential Computing documentation, Blackwell Architecture whitepaper, NVIDIA GPU Operator and Container Toolkit, and the NVIDIA Remote Attestation Service (NRAS). Relevant compliance references include NIST SP 800-207, the HIPAA Security Rule and GDPR Article 32.
