Oriel — AI & technology worth noticing, every day.

Benchmarking the Gemini Decisions API

Post · https://dev.to/gde/benchmarking-the-gemini-decisions-api-l31

The Gemini Decisions API scores a fixed set of answers in one pass and returns probabilities. An ADK triage agent built on it, a 48-ticket benchmark against Gemini 3.8 Flash, Gemini 4 Hydrogen, Gemini 3.5 Flash-Lite and Gemma 4 E2B, and a test of 1 to 100 questions in one request.

First seen by Oriel on .

AI summary

This tutorial benchmarks the Gemini Decisions API against three Gemini models and Gemma for customer-support triage, finding it fastest, inexpensive, confidence-aware, and competitive on fixed-set decisions.

  • The Decisions API scores up to 1,000 allowed-answer questions in one forward pass, returns probabilities, generates no text, and charges $0.10 per million input tokens.
  • Across 48 tickets, it achieved 46/48 team labels, 47/48 refund labels, and 29/48 tickets with all three answers correct.
  • Its 202 ms median latency was 3.5 times faster than Flash-Lite, 9.7 times faster than Hydrogen, and 11.7 times faster than 3.8 Flash.
  • A 0.7 confidence threshold identified all 17 urgency errors, enabling automatic handling of higher-confidence results and human review for uncertain ones.
  • The API answered 100 yes/no questions in 359 ms, while generative backends produced answers without confidence scores and scaled more slowly.

#gemini decisions api #customer support #model benchmarking #llm agents

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 21, latest observation ; +5 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 4, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
DEV Community5032399https://dev.to/gde/benchmarking-the-gemini-decisions-api-l31 Read discussion