I Retested My MoE Offloading Benchmarks. Some of My Conclusions Changed.
Post · https://dev.to/sizzlebop/i-retested-my-moe-offloading-benchmarks-some-of-my-conclusions-changed-6f4
A follow-up to my original 16GB GPU experiment, this time with consistent quantization, repeated...
First seen by Oriel on .
AI summary
This follow-up retests MoE offloading with consistent Q4_K_M quantization and repeated measurements, finding that quantization, placement, workload length, and speculative decoding substantially affect performance.
- The experiment used an RTX 4090 Laptop GPU with 16GB VRAM, llama.cpp, CUDA, Flash Attention, interleaved runs, warmups, cooldowns, and median throughput.
- On the same Qwen3.5-35B checkpoint, selective offloading was nearly tied at Q6_K but 29.5% faster than traditional splitting at Q4_K_M.
- LFM2.5-24B improved dramatically at Q4_K_M, while 512-token testing reduced both strategies to approximately 22 tok/s and eliminated the short-test advantage.
- Gemma 4’s MTP speculative decoding reduced throughput by approximately 46.7%, with only 11.68% of proposed draft tokens accepted.
- Run-to-run variation remained substantial, showing that three measurements and short generations can produce misleading performance conclusions.
#mixture-of-experts #llama.cpp #quantization #speculative decoding
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 35, latest observation ; +29 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 36, latest observation ; +35 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4954468 | https://dev.to/sizzlebop/i-retested-my-moe-offloading-benchmarks-some-of-my-conclusions-changed-6f4 Read discussion |