tencent/DiagSafe
Dataset · https://huggingface.co/datasets/tencent/DiagSafe
DiagSafe Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics Paper (arXiv) · GitHub · Evaluation code · Dashboard Content warning. This dataset contains harmful requests (e.g. violence, sexual content, self-harm, child safety, CBRN) written to test LLM safety. It is intended for safety evaluation and research only. DiagSafe is a diagnostic safety benchmark for large language models. It pairs 3,489 bilingual prompts, grounded in an 84-label… See the full description on the dataset page: https://huggingface.co/datasets/tencent/DiagSafe.
First seen by Oriel on .
AI summary
DiagSafe is a bilingual benchmark for diagnosing LLM safety with prompt-specific rubrics that require item-level judgments and evidence instead of binary labels.
- The dataset contains 3,489 prompts: 1,749 English and 1,740 Chinese, including 2,560 single-turn and 929 multi-turn examples.
- Prompts span 11 first-level, 34 second-level, and 84 third-level safety categories, covering areas including cybersecurity, CBRN, child safety, and misinformation.
- Each rubric includes 2–6 prompt-specific safety items with exemptions, eight usefulness items, a 0–3 usefulness score, and a JSON output contract.
- DiagAttack embeds harmful intent in benign-looking tasks using role, format, and tone constraints, while DiagBase varies language, dialogue length, and attack form.
- Rubric-based judging achieved 89.41%–95.34% agreement with human annotations across judges, but results depend on the judge model.
#llm safety #benchmarking #safety evaluation #llm judges
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Likes
Total likes reported by the source.
- Hugging Face Datasets: 9, latest observation ; 0 over observed period since
Downloads (30d)
Rolling 30-day downloads. A change in this value is not new downloads during the observed period.
- Hugging Face Datasets: 221, latest observation ; 0 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Datasets | tencent/DiagSafe | https://huggingface.co/datasets/tencent/DiagSafe Read discussion |