collinear-ai/cwe-bench
Dataset · https://huggingface.co/datasets/collinear-ai/cwe-bench
CWE-bench: Defensive Audit-and-Patch Tasks 100 defensive security tasks covering 91 distinct CWE types across 8 languages. In each task an agent is given a codebase containing one security weakness and must find and fix it without breaking the code's normal behavior. A fix is accepted only when the weakness is closed and legitimate behavior is preserved. Grading is fully executable: the reward comes from running exploit inputs and legitimate cases against the agent's patched… See the full description on the dataset page: https://huggingface.co/datasets/collinear-ai/cwe-bench.
First seen by Oriel on .
AI summary
CWE-bench is a benchmark of 100 defensive code-audit and patching tasks that scores agents on closing security weaknesses while preserving legitimate behavior.
- Tasks cover 91 CWE types across eight languages and use vulnerable codebases that agents must inspect and fix.
- A deterministic verifier runs exploit inputs and legitimate cases; a task earns reward 1 only when all checks pass, otherwise 0.
- The dataset uses Harbor tasks with Docker environments, reference fixes, and per-task instructions; agent runs have a 3,600-second limit.
- Access requires accepting repository conditions, and benchmark solutions must not be used to train evaluated models or republished.
- Results are tracked on Hugging Face and a leaderboard Space, with reported pass@1 averaged over four rollouts per task.
#cybersecurity #software security #benchmarking #llm agents
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Likes
Total likes reported by the source.
- Hugging Face Datasets: 15, latest observation ; +6 over observed period since
Downloads (30d)
Rolling 30-day downloads. A change in this value is not new downloads during the observed period.
- Hugging Face Datasets: 2,377, latest observation ; +1,879 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Datasets | collinear-ai/cwe-bench | https://huggingface.co/datasets/collinear-ai/cwe-bench Read discussion |