CWE-bench v2: More coverage, quality, and transparency
Web page · https://huggingface.co/blog/cwe-bench
First seen by Oriel on .
AI summary
CWE-bench v2 publishes 100 reproducible, programmatically graded security-fixing tasks across 91 CWEs, with a held-out set for tracking model progress.
- Each task gives an agent a real open-source repository and an instruction to fix a weakness while preserving existing functionality.
- Tasks use Dockerized Harbor environments and deterministic verifiers; the public dataset, grader, and harness are open, while access to the dataset is gated.
- Of 11 evaluated models, Gemini 4 Argon leads at 67.8% pass@1 and 81.0% pass@4; 95 tasks were solved by at least one model.
- The cost-performance Pareto frontier includes DeepSeek V4.1 Flash (1017) at 57.0% pass@1 for $0.04 per rollout, Claude Opus 5.5 at $1.83, and Gemini 4 Argon at $3.92.
- The environments and verifiers can also provide rewards for training defensive agents, not only for evaluation.
#cybersecurity #security benchmarks #llm agents #harbor
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Blog | https://huggingface.co/blog/cwe-bench | https://huggingface.co/blog/cwe-bench Read discussion |