Oriel — AI & technology worth noticing, every day.

Four Ways a Test Can Lie to You

Web page · https://kenwalger.com/blog/ai-engineering/four-ways-a-test-can-lie

Green and worthless: what verifying one small application taught me, when an AI agent wrote most of...

First seen by Oriel on .

AI summary

An account of an AI-assisted wine cellar project argues that green or red tests can mislead when controls, contracts, system boundaries, or coverage paths are wrong.

  • A proposed claim test was vacuous because the fixture already selected another claim; mutation testing exposed checks whose covered clauses could be deleted safely.
  • A credential scan always failed by matching skipCrossDatasetReferenceValidation, showing that checks need negative controls and must run consistently in CI.
  • A date-input test correctly verified clamping but encoded the wrong contract, because clamping during partial native-input edits overwrote the user’s year.
  • The application silently used production while authoring targeted staging, so internally consistent assertions validated a different dataset than intended.
  • Property testing across all 17,167 slider positions found an arithmetic bug missed by five carefully chosen demo dates; the author recommends precommitted oracles, mutation, measurement, and explicit not-verified outcomes.

#software testing #ai engineering #property testing #test oracles

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 52, latest observation ; +41 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 69, latest observation ; +65 over observed period since

Aliases

SourceExternal IDURL
DEV Community4878017https://www.kenwalger.com/blog/ai-engineering/four-ways-a-test-can-lie/ Read discussion