Oriel — AI & technology worth noticing, every day.

I Retested My MoE Offloading Benchmarks. Some of My Conclusions Changed.

Post · https://dev.to/sizzlebop/i-retested-my-moe-offloading-benchmarks-some-of-my-conclusions-changed-6f4

A follow-up to my original 16GB GPU experiment, this time with consistent quantization, repeated...

First seen by Oriel on .

AI summary

This follow-up retests MoE offloading with consistent Q4_K_M quantization and repeated measurements, finding that quantization, placement, workload length, and speculative decoding substantially affect performance.

  • The experiment used an RTX 4090 Laptop GPU with 16GB VRAM, llama.cpp, CUDA, Flash Attention, interleaved runs, warmups, cooldowns, and median throughput.
  • On the same Qwen3.5-35B checkpoint, selective offloading was nearly tied at Q6_K but 29.5% faster than traditional splitting at Q4_K_M.
  • LFM2.5-24B improved dramatically at Q4_K_M, while 512-token testing reduced both strategies to approximately 22 tok/s and eliminated the short-test advantage.
  • Gemma 4’s MTP speculative decoding reduced throughput by approximately 46.7%, with only 11.68% of proposed draft tokens accepted.
  • Run-to-run variation remained substantial, showing that three measurements and short generations can produce misleading performance conclusions.

#mixture-of-experts #llama.cpp #quantization #speculative decoding

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 35, latest observation ; +29 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 36, latest observation ; +35 over observed period since

Aliases

SourceExternal IDURL
DEV Community4954468https://dev.to/sizzlebop/i-retested-my-moe-offloading-benchmarks-some-of-my-conclusions-changed-6f4 Read discussion