MoE Offloading Isn’t Always Faster: I Tested 5 Models on a 16GB GPU
Post · https://dev.to/sizzlebop/moe-offloading-isnt-always-faster-i-tested-5-models-on-a-16gb-gpu-498a
I do a lot of experimenting with local language models, and I just ended up going down a bit of a...
First seen by Oriel on .
AI summary
Benchmarks of five MoE models on a 16GB RTX 4090 Laptop GPU show selective expert offloading can outperform layer offloading, but results vary sharply by model and configuration.
- On Phi-3.5-MoE and Apex-35B, selective offloading reached 5.67 and 5.18 tok/s, versus 4.03 and 3.69 tok/s with traditional offloading.
- Traditional offloading was faster for Qwen3.5-35B, Gemma 4 26B, and LFM2.5-24B-A2B; LFM reached 9.03 tok/s versus 2.11 tok/s selectively offloaded.
- For Phi-3.5-MoE, the fastest tested setup offloaded 20 expert layers to CPU, used 11,015 MiB VRAM, and generated 5.67 tok/s.
- The tests used llama.cpp with CUDA on a laptop with 32GB RAM; they were short benchmarks, and the author notes that more runs and profiling are needed to explain differences.
- Offloading all Phi-3.5-MoE experts caused heavy swapping, about 90MB/s, and reduced generation to around 3.14 tok/s.
#mixture of experts #llama.cpp #llm inference #gpu offloading
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 26, latest observation ; +8 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 18, latest observation ; +14 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4947837 | https://dev.to/sizzlebop/moe-offloading-isnt-always-faster-i-tested-5-models-on-a-16gb-gpu-498a Read discussion |