Oriel — AI & technology worth noticing, every day.

Three Times My Test Failed and the LLM Was Right

Post · https://dev.to/debashish_ghosal/three-times-my-test-failed-and-the-llm-was-right-43l2

A test fails, and the reflex is: the code is wrong. Debug the code, fix the code, move on. That...

First seen by Oriel on .

AI summary

This field-test report shows three agentwatch test failures caused by incorrect expectations, illustrating why LLM reasoning, floating-point behavior, and documented limitations must be checked before changing detector code.

  • GoalDriftDetector correctly treated search_kb as a prerequisite for resetting a password; replacing it with delete_database produced the intended divergence.
  • PerToolCostSpikeDetector correctly rejected a calculated dominance of 1.9999999999999998 against a 2.0 threshold; testing 7+2 avoids boundary ambiguity.
  • RetryStormDetector intentionally fires for total_retries >= 5, including successful transient retries, because this behavior is documented as a v0.1.0 blind spot.
  • EmbeddingDriftDetector shared baselines when scenarios reused agent_name, creating spurious comparisons; distinct names fixed the hidden state dependency.
  • The recommended debugging loop reads the model's reasoning, verifies its judgment, checks documentation, and changes the test or detector accordingly.

#llm testing #agent security #floating point #test design

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 25, latest observation ; +10 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 33, latest observation ; +22 over observed period since

Aliases

SourceExternal IDURL
DEV Community4796922https://dev.to/debashish_ghosal/three-times-my-test-failed-and-the-llm-was-right-43l2 Read discussion