Oriel — AI & technology worth noticing, every day.

Counting the Bandages: On measuring the time that AI saves

Paper · https://arxiv.org/abs/2610.35942

arXiv:2610.35942v1 Announce Type: new Abstract: I consider what it means for AI assistance to save time. I contrast a stopwatch view, on which time saved is whatever the clock records, with a ledger view, on which it is a counterfactual that has to be read next to the errors the work lets through, and I argue for the latter. The argument has four parts. I define a model-free pair of quantities, time saved and excess escaped defects, and one decision number built from them: the break-even defect cost. I give a baseline model of checking in which, until the model is good enough not to check, a calibrated worker's escaped-defect rate does not depend on how good the model is; checking time grows with the logarithm of the model's error rate; and over-trust buys minutes on the clock at a larger cost in total minutes. I show what randomized studies can and cannot identify, and I replicate METR's developer study from its public data: the same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them. Finally, I restate the published record, with stated bounds in place of imported point assumptions. In GDPval's own review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x; the consultants outside the frontier were faster, and more often wrong. Last, I test two of the model's predictions in a small pre-specified experiment on AI reviewers checking AI drafts of the same kind of brief, built from SEC filings made after the models' training data were collected. Told the drafter's true error rate, the reviewer checked more as it rose, but all-or-nothing, switching near where its prompt's costs break even. Told that the drafter erred ten times less often than it did, it checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none: faster and worse, entirely through reduced effort.

First seen by Oriel on .

AI summary

This paper argues that AI’s time savings should be measured against escaped errors, developing a defect-cost framework and applying it to developer, consulting, and AI-reviewer evidence.

  • It defines time saved, excess escaped defects, and break-even defect cost as model-free quantities for comparing assistance.
  • Its checking model predicts calibrated workers’ escaped-defect rate remains stable as model quality improves, while checking time grows logarithmically with error rate.
  • Reanalysis of METR’s 246-issue developer study finds a 19% or 24% slowdown depending on estimand, versus 34% unadjusted, with AI helping 3%–82% of issues.
  • GDPval’s review-every-output workflow limits GPT-5’s speedup to 3.6x; consultants outside the frontier were faster but more often wrong.
  • In a pre-specified AI-reviewer experiment, understated draft error rates reduced checking and caused wrong figures in 22.5% of briefs versus none when rates were accurately disclosed.

#ai evaluation #software engineering #productivity #human ai interaction

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.35942v1https://arxiv.org/abs/2610.35942 Read discussion