TensorRT Edge-LLM runs Qwen3.6-27B 6.4× faster than llama.cpp on Jetson AGX Thor in MLPerf Edge Agentic

NVIDIA submitted TensorRT Edge-LLM running Qwen3.6-27B on a single NVIDIA Jetson AGX Thor Developer Kit to the MLPerf Inference v6.1 Edge Agentic benchmark and reported a substantial speed improvement over the reference run. The system delivered 52.33 tokens per second and completed all 1,007 generated turns of the performance workload in 24 minutes 36 seconds — 6.4× faster than the llama.cpp reference submission on the same hardware, which finished in 2 hours 37 minutes.

What MLPerf Edge Agentic measures

MLPerf Edge Agentic evaluates an OpenAI-compatible model endpoint through two phases:

• Performance phase: recorded software-engineering agent trajectories are replayed. The model receives a user request, generates a tool call, observes the tool result, and continues the conversation. The workload covers 20 conversations and 1,007 generated turns; input length increases across turns and reaches roughly 23.5K tokens, so long-context processing is a key part of the measurement. IoU-based inline accuracy is also measured to ensure the agent is operating correctly during the performance run.
• Accuracy phase: uses Berkeley Function Calling Leaderboard (BFCL) v4 prompts in single-turn mode with reasoning off to balance accuracy evaluation time on edge devices. This phase checks whether the model selects the correct function, generates valid arguments, and refrains from calling a tool when none is required.

MLPerf results on Jetson AGX Thor

The TensorRT Edge-LLM submission ran Qwen3.6-27B in SingleStream mode on one Jetson AGX Thor Developer Kit with 128 GB unified memory and MAXN power mode. Reported metrics:

• Output throughput: 52.33 tokens per second
• Median time to first token: 247.12 ms
• Median time per output token: 14.68 ms
• BFCL overall accuracy: 87.94%

MLCommons also publishes a llama.cpp reference run on Jetson AGX Thor that uses Qwen3.6-27B with Q4_K_M quantization and completes in 2 hours 37 minutes. The TensorRT Edge-LLM submission completes the same performance workload in 24 minutes 36 seconds, or 6.4× less time.

Which optimizations enabled the speedup

The submission combines several techniques to accelerate inference while preserving the accuracy required by MLPerf:

• NVFP4 quantization: the submitted Qwen3.6-27B uses NVFP4 for weights and activations (including the language-model head) and FP8 for the KV cache. NVFP4 is a 4-bit floating-point format supported by the Blackwell GPU in Jetson AGX Thor. Shrinking model representation reduces DRAM bandwidth pressure and accelerates low-batch LLM decoding.

• KV-cache and recurrent-state reuse: in an agent trajectory each new request includes most of the prior conversation plus a new response or tool result. Without reuse, the shared history must be prefetched on every turn, growing cost as conversations lengthen. TensorRT Edge-LLM identifies reusable prompt prefixes and restores cached attention KV pages; for Qwen3.6’s hybrid architecture it also restores the recurrent state and partial KV-page state required to continue correctly. The runtime then only prefills the new suffix. On this workload about 96% of prompt tokens were served from hot cache; only ~0.5M of the total 13.6M prompt tokens were prefetched across the turns.

• Tree-based multi-token prediction (MTP): instead of standard one-token autoregressive decoding, multi-token prediction drafts several future tokens with a draft model and verifies them with the target model. TensorRT Edge-LLM implements a tree-based MTP where high-probability candidate continuations are organized into a tree and verified in one forward pass; matching paths are accepted, potentially advancing generation by several tokens. The MLPerf server config used 8 draft steps, top-2 candidates per drafting depth, and a 16-node verification tree. Tree-based verification is useful for function-calling because tool names, JSON syntax, and argument structures tend to be predictable. For this workload, the tree-based MTP could yield roughly an additional 40% decoding performance improvement compared with a linear MTP with 3 draft steps.

KV-cache reuse reduces repeated long-context computation before generation, while MTP lowers the number of target-model steps during generation; both optimizations act together to cut end-to-end runtime.

Reproducing the submission

The implementation used for the submission is available on the TensorRT Edge-LLM release/0.9.1-mlpinf branch. That branch includes model export settings, TensorRT engine build commands, server configuration, and MLPerf client configuration. High-level reproduction steps:

1. Clone TensorRT Edge-LLM and initialize its submodules.
2. Download the calibrated NVFP4 checkpoint (centml/Qwen3.6-27B-NVFP4-W4A4-mlpinf).
3. Follow mlperf/README.md to export the checkpoint with the tree-MTP interface and build base and draft TensorRT engines.
4. Launch the OpenAI-compatible TensorRT Edge-LLM server using the provided script.
5. Clone the MLCommons endpoint harness, install BFCL dependencies, set model and tokenizer paths in mlperf/config.yaml, and run the benchmark. The supplied config runs both performance and accuracy phases with temperature=0, seed=42, reasoning disabled, and concurrency=1; use "--accuracy-only" to run only the BFCL accuracy phase.

Acknowledgments

NVIDIA lists contributions from the TensorRT Edge-LLM, ModelOpt, Jetson, and MLPerf teams, notably Zihao Kong, Xiang Guo, Yoco Xiao, Qikai Li and Ashwin Nanjappa. MLCommons is credited for developing the Edge Agentic benchmark and endpoint harness.

---

For full MLPerf Inference v6.1 results across submissions see the MLCommons announcement; separate NVIDIA materials detail server-scale Vera Rubin NVL72 performance in the same benchmark cycle.