Oriel — AI & technology worth noticing, every day.

CWE-bench v2: More coverage, quality, and transparency

Web page · https://huggingface.co/blog/cwe-bench

First seen by Oriel on .

AI summary

CWE-bench v2 publishes 100 reproducible, programmatically graded security-fixing tasks across 91 CWEs, with a held-out set for tracking model progress.

  • Each task gives an agent a real open-source repository and an instruction to fix a weakness while preserving existing functionality.
  • Tasks use Dockerized Harbor environments and deterministic verifiers; the public dataset, grader, and harness are open, while access to the dataset is gated.
  • Of 11 evaluated models, Gemini 4 Argon leads at 67.8% pass@1 and 81.0% pass@4; 95 tasks were solved by at least one model.
  • The cost-performance Pareto frontier includes DeepSeek V4.1 Flash (1017) at 57.0% pass@1 for $0.04 per rollout, Claude Opus 5.5 at $1.83, and Gemini 4 Argon at $3.92.
  • The environments and verifiers can also provide rewards for training defensive agents, not only for evaluation.

#cybersecurity #security benchmarks #llm agents #harbor

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
Hugging Face Bloghttps://huggingface.co/blog/cwe-benchhttps://huggingface.co/blog/cwe-bench Read discussion