Four Ways a Test Can Lie to You
Web page · https://kenwalger.com/blog/ai-engineering/four-ways-a-test-can-lie
Green and worthless: what verifying one small application taught me, when an AI agent wrote most of...
First seen by Oriel on .
AI summary
An account of an AI-assisted wine cellar project argues that green or red tests can mislead when controls, contracts, system boundaries, or coverage paths are wrong.
- A proposed claim test was vacuous because the fixture already selected another claim; mutation testing exposed checks whose covered clauses could be deleted safely.
- A credential scan always failed by matching skipCrossDatasetReferenceValidation, showing that checks need negative controls and must run consistently in CI.
- A date-input test correctly verified clamping but encoded the wrong contract, because clamping during partial native-input edits overwrote the user’s year.
- The application silently used production while authoring targeted staging, so internally consistent assertions validated a different dataset than intended.
- Property testing across all 17,167 slider positions found an arithmetic bug missed by five carefully chosen demo dates; the author recommends precommitted oracles, mutation, measurement, and explicit not-verified outcomes.
#software testing #ai engineering #property testing #test oracles
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 52, latest observation ; +41 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 69, latest observation ; +65 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4878017 | https://www.kenwalger.com/blog/ai-engineering/four-ways-a-test-can-lie/ Read discussion |