AI agent fixed the bug, but what else did it change?
Web page · https://blog.trailofbits.com/2026/10/21/your-ai-agent-fixed-the-bug-but-what-else-did-it-change
First seen by Oriel on .
AI summary
Trail of Bits introduces patching-bugs and regression-hunt, plugins that structure agent-led bug fixes and live proof-based regression checks; preliminary tests found 25% fewer false positives than a baseline Codex agent.
- patching-bugs reproduces reported bugs, develops fixes, and uses a three-arm proof: the original and reverted code must fail while the patched code passes.
- regression-hunt traces code changes to real interfaces, reproduces candidates on live deployments, and uses a fresh-context agent plus adjudication to validate evidence.
- Across 21 confirmed regressions, regression-hunt reduced false positives by 25%, with a 95% confidence interval of 14% to 38%.
- Strict evidence requirements left some known regressions borderline; the evaluation also found 21 regressions among more than 130 merged PRs, and none broke a security protection.
- Maintainers can guide the plugins with THREAT_MODEL.md and PATCHING_GUIDELINES.md or AGENTS.md; both plugins provide modular steps and defined input and output schemas.
#ai agents #software security #regression testing #open source
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 49, latest observation ; +42 over observed period since
- Lobsters: 2, latest observation ; +2 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 6, latest observation ; +5 over observed period since
- Lobsters: 0, latest observation ; 0 over observed period since