Postmortem: The Tuesday Our AI Bill Had No Owner
Post · https://dev.to/gde/postmortem-the-tuesday-our-ai-bill-had-no-owner-34pd
I've read a lot of postmortems. The best ones aren't about blame. They're about the gap between what...
First seen by Oriel on .
AI summary
This composite postmortem traces a threefold Vertex AI cost spike to an unbounded review retry loop and argues that per-request labels would have made the cause identifiable in minutes.
- A nightly ticket-summarization workflow ran six hours instead of its usual forty minutes because repeated rejections expanded each rewrite's context; the output was eventually correct and customers were unaffected.
- Shared project billing lacked service-, workflow-, and step-level attribution, extending root-cause analysis to 2 hours 14 minutes; detection came about eight hours after the spike began.
- The proposed Vertex AI request labels identify dimensions such as team, use_case, agent, step, and prompt_version, and flow into Cloud Billing reports and BigQuery exports on PayGo.
- Recommended controls include a labeled client wrapper, label propagation across agents, CI enforcement, cost-coverage queries, real-time token alerts, and retry caps.
- The article cautions against high-cardinality or sensitive label values and notes limits of 64 labels for Google models and 32 for partner models; direct Gemini API keys do not support this field.
#vertex ai #cloud billing #llm observability #incident management
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 18, latest observation ; +4 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 13, latest observation ; +7 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4915028 | https://dev.to/gde/postmortem-the-tuesday-our-ai-bill-had-no-owner-34pd Read discussion |