Nothing Reported It
Web page · https://kenwalger.com/blog/ai/nothing-reported-it
Five ways my own test harness lied to me, and the one that nearly got published The number...
First seen by Oriel on .
AI summary
The post documents five test-harness flaws that produced plausible but invalid model-evaluation results, including missing inputs, misleading metrics, undetected measurement errors, and misdiagnosed warnings.
- A missing corpus caused grounded QA prompts to omit all reference material, making abstention scores and conclusions about small language models meaningless.
- Metrics rewarded indiscriminate escalation or abstention, while keyword checks produced false negatives and false positives, including an answer with an incorrect fee.
- A one-directional latency detector missed implausible speedups, and failures were initially excluded from scoring even when they represented genuine model failures.
- The revised harness distinguishes model failures from transport failures, rejects empty or implausibly small corpora, reports corpus size and prompt estimates, and verifies generated artifacts.
- The author argues that reproducibility can faithfully reproduce a harness bug, so evaluation checks must be derived from observed failures and validated against output data.
#model evaluation #test harnesses #grounded qa #ollama
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 20, latest observation ; +1 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 21, latest observation ; +8 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4839625 | https://www.kenwalger.com/blog/ai/nothing-reported-it Read discussion |