Run AI Models for Edge Application Development with Just a Few Commands
AI is expanding beyond the data center as developers build a new generation of agents to act at the edge: monitoring driver attention and cabin conditions,...
First seen by Oriel on .
AI summary
The tutorial shows how to run NVIDIA Nemotron-3.5 Lightning on DGX Spark with TensorRT Edge-LLM, serve it through an OpenAI-compatible API, and optionally accelerate generation with DSpark.
- The `trtedge` CLI downloads the model and builds TensorRT engines; the first setup requires about 21GB for the model, plus space for engines.
- The tutorial requires DGX Spark, NVIDIA CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series, and Python 3.10 to 3.12.
- On DGX Spark 64GB, DSpark with three draft tokens raised mean generation speed from 73.3 to 108.5 tokens/s on SpeedBench 1K, while median time to first token was 203.0 ms.
- Peak system memory during DSpark inference was 27.9GB; measurements used batch size one, one concurrent streaming request, and a 128-token output limit.
#tensorrt edge-llm #dgx spark #speculative decoding #edge ai
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| NVIDIA Developer Blog | https://developer.nvidia.com/blog/?p=124453 | https://developer.nvidia.com/blog/run-ai-models-for-edge-application-development-with-just-a-few-commands/ Read discussion |