High or Low Thinking? Gemini 4 and Gemma 4 on Hard Math, Cost First
Post · https://dev.to/gde/high-or-low-thinking-gemini-4-and-gemma-4-on-hard-math-cost-first-3091
Seven models on the 15 problems of AIME 2025 I at their lowest and highest thinking settings, under a hard budget. The top setting cost 2.6 to 4.8 times as much on every paid model; Hydrogen, Carbon and Argon solved all 15 at either setting, and no score change was large enough to rule out chance on 15 problems.
First seen by Oriel on .
AI summary
This benchmark compares seven models at minimum and maximum thinking settings on 15 AIME 2025 problems, finding higher thinking greatly increased cost and latency without improving Gemini 4 accuracy.
- Hydrogen, Carbon, and Argon each solved all 15 problems at both settings; smaller models improved by one or two problems, a difference compatible with chance.
- The two runs cost $1.53 at the lowest settings and $5.95 at the highest, while paid-model median time rose 3.2 to 5.9 times.
- Argon at its lowest setting delivered the cheapest correct answer at $0.0162; highest-setting costs ranged from 2.61 to 4.75 times lower-setting costs.
- With only 15 problems and one run, the study cannot distinguish close scores and leaves Carbon’s proposed long multi-step workloads untested.
#gemini #gemma #model evaluation #math benchmarks
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 24, latest observation ; +5 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 5, latest observation ; +2 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 5043905 | https://dev.to/gde/high-or-low-thinking-gemini-4-and-gemma-4-on-hard-math-cost-first-3091 Read discussion |