Oriel — AI & technology worth noticing, every day.

Idavidrein/gpqa

Dataset · https://huggingface.co/datasets/Idavidrein/gpqa

Dataset Card for GPQA GPQA is a multiple-choice, Q&A dataset of very hard questions written and validated by experts in biology, physics, and chemistry. When attempting questions out of their own domain (e.g., a physicist answers a chemistry question), these experts get only 34% accuracy, despite spending >30m with full access to Google. We request that you do not reveal examples from this dataset in plain text or images online, to reduce the risk of leakage into foundation… See the full description on the dataset page: https://huggingface.co/datasets/Idavidrein/gpqa.

First seen by Oriel on .

AI summary

GPQA is a 448-question expert-authored benchmark designed to test difficult scientific reasoning and support scalable oversight experiments.

  • Questions cover biology, physics, and chemistry and were written and validated by domain experts.
  • Experts with relevant PhDs achieved 65% accuracy, or 74% after discounting retrospectively identified mistakes.
  • Skilled non-experts achieved 34% accuracy after spending over 30 minutes per question with unrestricted web access.
  • A strongest GPT-4 based baseline achieved 39% accuracy, indicating difficulty for frontier AI systems.
  • The dataset is available under CC BY 4.0 and users must accept conditions prohibiting public disclosure of examples.

#gpqa #llm benchmarking #scalable oversight #scientific reasoning

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Likes

Total likes reported by the source.

  • Hugging Face Datasets: 644, latest observation ; +53 over observed period since

Downloads (30d)

Rolling 30-day downloads. A change in this value is not new downloads during the observed period.

  • Hugging Face Datasets: 104,598, latest observation ; +1,901 over observed period since

Aliases

SourceExternal IDURL
Hugging Face DatasetsIdavidrein/gpqahttps://huggingface.co/datasets/Idavidrein/gpqa Read discussion