Oriel — AI & technology worth noticing, every day.

collinear-ai/cwe-bench

Dataset · https://huggingface.co/datasets/collinear-ai/cwe-bench

CWE-bench: Defensive Audit-and-Patch Tasks 100 defensive security tasks covering 91 distinct CWE types across 8 languages. In each task an agent is given a codebase containing one security weakness and must find and fix it without breaking the code's normal behavior. A fix is accepted only when the weakness is closed and legitimate behavior is preserved. Grading is fully executable: the reward comes from running exploit inputs and legitimate cases against the agent's patched… See the full description on the dataset page: https://huggingface.co/datasets/collinear-ai/cwe-bench.

First seen by Oriel on .

AI summary

CWE-bench is a benchmark of 100 defensive code-audit and patching tasks that scores agents on closing security weaknesses while preserving legitimate behavior.

  • Tasks cover 91 CWE types across eight languages and use vulnerable codebases that agents must inspect and fix.
  • A deterministic verifier runs exploit inputs and legitimate cases; a task earns reward 1 only when all checks pass, otherwise 0.
  • The dataset uses Harbor tasks with Docker environments, reference fixes, and per-task instructions; agent runs have a 3,600-second limit.
  • Access requires accepting repository conditions, and benchmark solutions must not be used to train evaluated models or republished.
  • Results are tracked on Hugging Face and a leaderboard Space, with reported pass@1 averaged over four rollouts per task.

#cybersecurity #software security #benchmarking #llm agents

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Likes

Total likes reported by the source.

  • Hugging Face Datasets: 15, latest observation ; +6 over observed period since

Downloads (30d)

Rolling 30-day downloads. A change in this value is not new downloads during the observed period.

  • Hugging Face Datasets: 2,377, latest observation ; +1,879 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Datasetscollinear-ai/cwe-benchhttps://huggingface.co/datasets/collinear-ai/cwe-bench Read discussion