Compact open models released in 2026 bring multi-step reasoning and agentic capabilities that previously required large data-center systems to NVIDIA Jetson devices. Running these models locally reduces network dependency, lowers cost, and keeps sensitive data on device, enabling use cases such as in-cab assistants, real-time anomaly detection, and robots operating in remote or connectivity-limited environments.
What changed and why it matters
Improvements in training methods and more efficient architectures (for example, distillation) have produced 2026 models whose intelligence scores approach 2025 frontier models while using far fewer parameters. This shift makes it feasible to run reasoning-capable models at the edge on NVIDIA Jetson hardware, improving latency and reliability for on-site agent workflows.
Choosing a reasoning model for Jetson
Architecture and training create tradeoffs among capability, memory use, and generation speed. Two illustrative models:
- Qwen3.8-27B: a dense model that activates all 27 billion parameters per token; suitable for tasks that can tolerate longer per-token computation and favor higher-quality token decisions.
- Nemotron 3.5 Lightning: a mixture-of-experts (MoE) model with 30 billion total parameters but only ~3 billion active per token; this yields faster token generation for response-heavy workloads.
For long-running agents that monitor sensors and logs, take corrective actions, run validation tests and escalate only when needed, Nemotron 3.5 Lightning often shortens the overall loop time. Qwen3.8-27B is often better for workloads that need fewer but harder decisions and allow more generation time per response. Benchmark both models on the actual decisions, tools and response patterns your application needs before choosing.
On Jetson, agent loops run next to the sensors they interact with. Models can be deployed locally with frameworks such as vLLM and llama.cpp so that reasoning does not require continuous data-center access. Hardware mapping:
- Jetson Orin Nano: Gemma 4 E4B is a strong starting point.
- Jetson AGX Orin and Jetson AGX Thor: Nemotron 3.5 Lightning and Qwen3.8-27B are strong options with high-quality quantized checkpoints and optimized deployment choices.
How to optimize inference on Jetson
Two complementary techniques improve inference performance:
- NVFP4 quantization reduces compute and memory transfer per model pass by using lower-precision representations while keeping quality close to BF16.
- Speculative decoding uses a smaller draft model to propose multiple tokens that the main model verifies together; when multiple tokens are accepted, the generation advances several tokens in a single verification step.
Model architecture determines the baseline, but serving choices also matter. In tests, combining NVFP4 quantization with the best speculative decoding configuration for each model produced up to 6.28x decode throughput compared to BF16 on Jetson.
Speculative decoding methods include MTP, DFlash and DSpark; all run on Jetson but produce and evaluate drafts differently. The best method differed per model in tests: Nemotron 3.5 Lightning performed best with DSpark, while Qwen3.8-27B performed best with DFlash2. Always test the method and draft checkpoint with the model you plan to deploy rather than assuming one configuration fits all.
Measured throughput by workload
Using a fixed fastest configuration for each model, four SpeedBench categories were tested: writing, reasoning, summarization, and retrieval-augmented generation (RAG). Results:
- Nemotron 3.5 Lightning + DSpark (NVFP4): 123.01–138.02 output tokens/s depending on workload.
- Qwen3.8-27B + DFlash2 (NVFP4): 27.69–34.44 output tokens/s depending on workload.
Speculative decoding speedup varied by workload; writing and RAG saw the largest gains for both models. Validate speculative configurations using prompts representative of your application, as acceptance rate of draft tokens determines realized acceleration.
Prerequisites and example vLLM deployment
Prerequisites before running the examples:
- Jetson AGX Thor or Jetson AGX Orin (Gemma 4 E4B is a good start for Jetson Orin Nano)
- JetPack 7.2 with NVIDIA Container Runtime and Docker configured
- Enough storage for model and draft checkpoints
- Accepted license terms for NVIDIA Nemotron and Qwen3.8 checkpoints
Start the vllm container:
docker run --pull=always --runtime nvidia --rm -it \
--network host \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash \
vllm/vllm-openai:v0.28.0
Nemotron 3.5 Lightning (NVFP4 + DSpark) example inside the container:
vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
--reasoning-parser nemotron_v3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--max-model-len 128000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.7 \
--trust-remote-code \
--max-num-batched-tokens 16384 \
--enable-prefix-caching \
--speculative-config '{"method":"dspark","model":"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark","num_speculative_tokens":5}' \
--mamba-backend flashinfer \
--mamba-ssm-cache-dtype float16 \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--mamba-cache-mode align
Qwen3.8-27B (NVFP4 + DFlash2) example:
VLLM_GDN_DECODE_KERNEL=triton vllm serve Inferact/Qwen3.8-27B-NVFP4 \
--served-model-name qwen38 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--max-model-len 50000 \
--max-num-seqs 8 \
--gpu-memory-utilization 0.85 \
--trust-remote-code \
--speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'
Each speculative method has different cost/accuracy tradeoffs; MTP uses prediction heads trained with the main model, DFlash uses a diffusion-based draft model, and DSpark extends DFlash with draft corrections and early stopping.
When to train custom checkpoints
Start with existing quantized checkpoints and draft models; they typically provide strong accuracy and speedups without additional training. If quantization degrades accuracy for your specialized workload, use NVIDIA Model Optimizer to perform quantization-aware training (QAT) or quantization-aware distillation (QAD). If draft-token acceptance rates are too low, train a speculator via the vLLM Speculators training guide using representative application data. Speculators supports methods including MTP, EAGLE-3, DFlash and DSpark.
Most projects will not require custom training: measure accuracy and performance on your prompts first and only train when there is a clear gap.
Validation and final notes
Validate both throughput (output tokens/s) and draft-token acceptance rate on representative prompts to understand real-world speedup and accuracy tradeoffs. With optimized runtimes, quantized checkpoints and speculative decoding, Jetson enables moving from model evaluation to building and shipping edge inference applications.
For more detailed recipes, benchmarks and tutorials (running LLMs/VLMs on Jetson, benchmarking generative models, and getting started with speculative decoding), consult the Jetson AI Lab Models resources.
