Oriel — AI & technology worth noticing, every day.

MoE Offloading Isn’t Always Faster: I Tested 5 Models on a 16GB GPU

Post · https://dev.to/sizzlebop/moe-offloading-isnt-always-faster-i-tested-5-models-on-a-16gb-gpu-498a

I do a lot of experimenting with local language models, and I just ended up going down a bit of a...

First seen by Oriel on .

AI summary

Benchmarks of five MoE models on a 16GB RTX 4090 Laptop GPU show selective expert offloading can outperform layer offloading, but results vary sharply by model and configuration.

  • On Phi-3.5-MoE and Apex-35B, selective offloading reached 5.67 and 5.18 tok/s, versus 4.03 and 3.69 tok/s with traditional offloading.
  • Traditional offloading was faster for Qwen3.5-35B, Gemma 4 26B, and LFM2.5-24B-A2B; LFM reached 9.03 tok/s versus 2.11 tok/s selectively offloaded.
  • For Phi-3.5-MoE, the fastest tested setup offloaded 20 expert layers to CPU, used 11,015 MiB VRAM, and generated 5.67 tok/s.
  • The tests used llama.cpp with CUDA on a laptop with 32GB RAM; they were short benchmarks, and the author notes that more runs and profiling are needed to explain differences.
  • Offloading all Phi-3.5-MoE experts caused heavy swapping, about 90MB/s, and reduced generation to around 3.14 tok/s.

#mixture of experts #llama.cpp #llm inference #gpu offloading

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 26, latest observation ; +8 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 18, latest observation ; +14 over observed period since

Aliases

SourceExternal IDURL
DEV Community4947837https://dev.to/sizzlebop/moe-offloading-isnt-always-faster-i-tested-5-models-on-a-16gb-gpu-498a Read discussion