Oriel — AI & technology worth noticing, every day.

Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K

Post · https://dev.to/gde/accuracy-vs-speed-gemma-4-e2b-quantization-tradeoffs-bf16-to-q2k-aea

Gemma 4 E2B quantized from BF16 to Q2_K on one Tesla T4, every rung asked the same 250 GSM8K questions, and each answer buried at the rung where it first goes wrong. Q5_K_M and above hold the score; Q3_K_M loses 38; Q2_K gets 3 right. With thinking on, Q3_K_M wins back 17.

First seen by Oriel on .

AI summary

This article benchmarks nine Gemma 4 E2B GGUF quantization levels with llama.cpp on 250 maths questions, finding Q5_K_M the best accuracy-speed trade-off on a Tesla T4.

  • Using BF16's 229 correct answers as reference, Q5_K_M achieved 231, while Q4_0 reached 222 and Q2_K only 3.
  • Q5_K_M used 39.0% of BF16 bytes and decoded at 78.7 tok/s, while fastest Q4_0 reached 93.6 tok/s.
  • The graveyard records each first failure as a tombstone and tracks later recoveries, exposing 226 tombstones and 11 ghosts across the ladder.
  • Q3_K_M lost 42 answers, mostly because 34 deaths were cut off at 1,024 tokens; Q2_K had 246 cutoffs and 228 loops.
  • Google's QAT_Q4_0 scored 221 versus plain Q4_0's 222, with 15 deaths each but only six shared questions.

#gemma #gguf #llama.cpp #quantization

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 11, latest observation ; +3 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 10, latest observation ; +8 over observed period since

Aliases

SourceExternal IDURL
DEV Community4935418https://dev.to/gde/accuracy-vs-speed-gemma-4-e2b-quantization-tradeoffs-bf16-to-q2k-aea Read discussion