3 Ways to Boost MoE Inference on NVIDIA DGX Spark
Web page · https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark
Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA...
First seen by Oriel on .
AI summary
The post shows that combining Humming MXFP4 kernels, online FP8 LM-head quantization, and NEXTN speculative decoding increased Ling-3.0-flash decode throughput by 42.5% on DGX Spark across GSM8K, with no statistically detectable accuracy change.
- Humming specializes MXFP4 MoE kernels at runtime and improved synthetic-workload throughput by 9.3% over the baseline.
- Online FP8 quantization halves LM-head weight reads; together with Humming, it raised synthetic throughput 21.7% over baseline.
- NEXTN drafts three tokens and verifies four positions; GSM8K accept length averaged 2.68 tokens per round.
- Across 1,319 GSM8K questions, mean latency fell 24.5%, while accuracy changed from 96.89% to 96.82%.
- Combining FP8 with MTP requires sharing the full LM-head module; otherwise draft weights are misinterpreted, acceptance drops to 1.00, and throughput can fall below baseline.
#mixture of experts #fp8 quantization #speculative decoding #inference optimization
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| NVIDIA Developer Blog | https://developer.nvidia.com/blog/?p=124221 | https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark/ Read discussion |