Oriel — AI & technology worth noticing, every day.

High or Low Thinking? Gemini 4 and Gemma 4 on Hard Math, Cost First

Post · https://dev.to/gde/high-or-low-thinking-gemini-4-and-gemma-4-on-hard-math-cost-first-3091

Seven models on the 15 problems of AIME 2025 I at their lowest and highest thinking settings, under a hard budget. The top setting cost 2.6 to 4.8 times as much on every paid model; Hydrogen, Carbon and Argon solved all 15 at either setting, and no score change was large enough to rule out chance on 15 problems.

First seen by Oriel on .

AI summary

This benchmark compares seven models at minimum and maximum thinking settings on 15 AIME 2025 problems, finding higher thinking greatly increased cost and latency without improving Gemini 4 accuracy.

  • Hydrogen, Carbon, and Argon each solved all 15 problems at both settings; smaller models improved by one or two problems, a difference compatible with chance.
  • The two runs cost $1.53 at the lowest settings and $5.95 at the highest, while paid-model median time rose 3.2 to 5.9 times.
  • Argon at its lowest setting delivered the cheapest correct answer at $0.0162; highest-setting costs ranged from 2.61 to 4.75 times lower-setting costs.
  • With only 15 problems and one run, the study cannot distinguish close scores and leaves Carbon’s proposed long multi-step workloads untested.

#gemini #gemma #model evaluation #math benchmarks

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 24, latest observation ; +5 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 5, latest observation ; +2 over observed period since

Aliases

SourceExternal IDURL
DEV Community5043905https://dev.to/gde/high-or-low-thinking-gemini-4-and-gemma-4-on-hard-math-cost-first-3091 Read discussion