Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K
Post · https://dev.to/gde/accuracy-vs-speed-gemma-4-e2b-quantization-tradeoffs-bf16-to-q2k-aea
Gemma 4 E2B quantized from BF16 to Q2_K on one Tesla T4, every rung asked the same 250 GSM8K questions, and each answer buried at the rung where it first goes wrong. Q5_K_M and above hold the score; Q3_K_M loses 38; Q2_K gets 3 right. With thinking on, Q3_K_M wins back 17.
First seen by Oriel on .
AI summary
This article benchmarks nine Gemma 4 E2B GGUF quantization levels with llama.cpp on 250 maths questions, finding Q5_K_M the best accuracy-speed trade-off on a Tesla T4.
- Using BF16's 229 correct answers as reference, Q5_K_M achieved 231, while Q4_0 reached 222 and Q2_K only 3.
- Q5_K_M used 39.0% of BF16 bytes and decoded at 78.7 tok/s, while fastest Q4_0 reached 93.6 tok/s.
- The graveyard records each first failure as a tombstone and tracks later recoveries, exposing 226 tombstones and 11 ghosts across the ladder.
- Q3_K_M lost 42 answers, mostly because 34 deaths were cut off at 1,024 tokens; Q2_K had 246 cutoffs and 228 loops.
- Google's QAT_Q4_0 scored 221 versus plain Q4_0's 222, with 15 deaths each but only six shared questions.
#gemma #gguf #llama.cpp #quantization
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 11, latest observation ; +3 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 10, latest observation ; +8 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4935418 | https://dev.to/gde/accuracy-vs-speed-gemma-4-e2b-quantization-tradeoffs-bf16-to-q2k-aea Read discussion |