Oriel — AI & technology worth noticing, every day.

3 Ways to Boost MoE Inference on NVIDIA DGX Spark

Web page · https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark

Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA...

First seen by Oriel on .

AI summary

The post shows that combining Humming MXFP4 kernels, online FP8 LM-head quantization, and NEXTN speculative decoding increased Ling-3.0-flash decode throughput by 42.5% on DGX Spark across GSM8K, with no statistically detectable accuracy change.

  • Humming specializes MXFP4 MoE kernels at runtime and improved synthetic-workload throughput by 9.3% over the baseline.
  • Online FP8 quantization halves LM-head weight reads; together with Humming, it raised synthetic throughput 21.7% over baseline.
  • NEXTN drafts three tokens and verifies four positions; GSM8K accept length averaged 2.68 tokens per round.
  • Across 1,319 GSM8K questions, mean latency fell 24.5%, while accuracy changed from 96.89% to 96.82%.
  • Combining FP8 with MTP requires sharing the full LM-head module; otherwise draft weights are misinterpreted, acceptance drops to 1.00, and throughput can fall below baseline.

#mixture of experts #fp8 quantization #speculative decoding #inference optimization

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=124221https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark/ Read discussion