NVIDIA shows how developers can prototype and serve embedded AI models on DGX Spark configured with 64GB using TensorRT Edge-LLM. The tutorial walks through installing the runtime, launching NVIDIA Nemotron-3.5 Lightning (about 21 GB), hosting it via an OpenAI-compatible local API, and optionally enabling DSpark speculative decoding to accelerate token generation. Measured performance for time-to-first-token and tokens/second with and without DSpark on DGX Spark 64GB is included.
Why this matters
Physical edge AI applications — for example monitoring driver attention and cabin conditions, helping robots perceive and plan tasks, or coordinating machines on production lines — must operate within strict memory, power, and latency constraints. TensorRT Edge-LLM together with DGX Spark 64GB gives developers a platform to prototype, fine-tune, and test models and agents on a development system, while using a common inference framework for deployment to edge platforms such as NVIDIA Jetson Thor, NVIDIA DRIVE AGX Thor, and NVIDIA IGX Thor.
Prerequisites and environment
Requirements for the examples in this article:
- DGX Spark 64GB with a supported NVIDIA software stack
- NVIDIA CUDA 13
- TensorRT 10.16.1.11 or newer within the 10.x series
- Python 3.10–3.12
The examples were tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11.
Installation and environment setup
On your DGX Spark open a terminal, create and activate a virtual environment, then install TensorRT Edge-LLM:
python3 -m venv --system-site-packages .venv-edgellm source .venv-edgellm/bin/activate python -m pip install --upgrade pip python -m pip install "tensorrt-edgellm[server]==0.11.1"
The virtual environment keeps newly installed packages separate while allowing access to system Python packages, including TensorRT.
Verify the TensorRT version and that the TensorRT Edge-LLM runtime loads:
python -c "import tensorrt as trt; v = tuple(map(int, trt.version.split('.'))); assert (10, 16, 1, 11) <= v < (11,), f'TensorRT >=10.16.1.11 and <11 required; found {trt.version}'; from tensorrt_edgellm import runtime; runtime.load()"
If compatible system TensorRT bindings are not available, follow the NVIDIA TensorRT pip installation instructions to install a compatible TensorRT 10.x version in the activated environment, then rerun the verification command.
Start a chat with Nemotron-3.5 Lightning
With the virtual environment active, start a chat session with the model:
trtedge chat nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
On first launch the model download (approximately 21 GB) and engine build will complete; files are stored by default in ~/.cache/tensorrt-edgellm, so ensure enough disk space for both the model and engines. Later launches reuse matching cached engines.
When the chat prompt appears, try:
Convert 25°C to Fahrenheit. Answer in one sentence.
Example response: 25°C is equal to 77°F.
Continue the conversation or type /exit to end the session.
Serve the model via an OpenAI-compatible API
To develop applications against a local API, start the server with the virtual environment active:
trtedge serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
--host 127.0.0.1
--port 8000
The server builds or reuses an engine for this configuration. In a second terminal check readiness:
curl -fsS http://127.0.0.1:8000/health/ready
A response containing "status":"ready" confirms the server is ready. Send a prompt to the chat completions endpoint:
curl -fsS -N http://127.0.0.1:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
"messages": [
{
"role": "user",
"content": "Convert 25°C to Fahrenheit. Answer in one sentence."
}
],
"stream": true,
"temperature": 0,
"max_tokens": 128
}'
The response streams as JSON events. Keep the server running for more requests, or press Ctrl+C in the server terminal to stop it.
Optional: Enable DSpark speculative decoding
You can speed up generation using DSpark speculative decoding, which uses a draft model to propose multiple tokens that the main model verifies in a single pass. To enable three draft tokens, stop the server (Ctrl+C) and restart it with DSpark enabled:
trtedge serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
--host 127.0.0.1
--port 8000
--speculative-config '{
"method": "dspark",
"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark",
"num_speculative_tokens": 3
}'
After the server is ready, send another prompt to try DSpark-enabled generation. Experiment with num_speculative_tokens and other settings to tune generation speed for your workload.
Performance on DGX Spark 64GB
TensorRT Edge-LLM delivers low response latency and fast token generation for NVIDIA Nemotron-3.5 Lightning on DGX Spark 64GB. Reported median time-to-first-token (TTFT) and mean generation speed (tokens/s):
- DSpark OFF, SpeedBench 1K: median TTFT 201.0 ms, speed 73.3 tokens/s
- DSpark OFF, SpeedBench 2K: median TTFT 329.5 ms, speed 72.8 tokens/s
- DSpark OFF, HumanEval prompts: median TTFT 98.1 ms, speed 73.4 tokens/s
- DSpark ON, SpeedBench 1K: median TTFT 203.0 ms, speed 108.5 tokens/s
- DSpark ON, SpeedBench 2K: median TTFT 331.6 ms, speed 105.3 tokens/s
- DSpark ON, HumanEval prompts: median TTFT 98.7 ms, speed 126.9 tokens/s
Notes on the measurements:
- Generation speed is the average token generation rate per request, excluding time to first token.
- DSpark used three draft tokens in the measurements.
- Peak total system memory use during inference with DSpark was 27.9 GB, including the OS.
- Measurements were taken on DGX Spark 64GB running DGX OS 7.6.0 (Ubuntu 24.04.4 LTS), using NVFP4 with thinking enabled, batch size 1, one concurrent streaming request, and a 128-token output limit. Average input lengths: SpeedBench 1K 1,153 tokens, SpeedBench 2K 2,247 tokens, HumanEval 139 tokens.
Next steps
Try other supported models and continue developing your application on Spark. Options include vision-language models that interpret images and video, as well as speech recognition and text-to-speech models. Consult the TensorRT Edge-LLM documentation for model-specific workflows and requirements on your target embedded platform, visit the Nemotron developer page for resources, and explore open Nemotron models and datasets on Hugging Face and the Blueprints available on build.nvidia.com.
