Three Times My Test Failed and the LLM Was Right
Post · https://dev.to/debashish_ghosal/three-times-my-test-failed-and-the-llm-was-right-43l2
A test fails, and the reflex is: the code is wrong. Debug the code, fix the code, move on. That...
First seen by Oriel on .
AI summary
This field-test report shows three agentwatch test failures caused by incorrect expectations, illustrating why LLM reasoning, floating-point behavior, and documented limitations must be checked before changing detector code.
- GoalDriftDetector correctly treated search_kb as a prerequisite for resetting a password; replacing it with delete_database produced the intended divergence.
- PerToolCostSpikeDetector correctly rejected a calculated dominance of 1.9999999999999998 against a 2.0 threshold; testing 7+2 avoids boundary ambiguity.
- RetryStormDetector intentionally fires for total_retries >= 5, including successful transient retries, because this behavior is documented as a v0.1.0 blind spot.
- EmbeddingDriftDetector shared baselines when scenarios reused agent_name, creating spurious comparisons; distinct names fixed the hidden state dependency.
- The recommended debugging loop reads the model's reasoning, verifies its judgment, checks documentation, and changes the test or detector accordingly.
#llm testing #agent security #floating point #test design
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 25, latest observation ; +10 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 33, latest observation ; +22 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4796922 | https://dev.to/debashish_ghosal/three-times-my-test-failed-and-the-llm-was-right-43l2 Read discussion |