Oriel — AI & technology worth noticing, every day.

Can we trust LLMs to judge whether a research idea is novel?

Web page · https://huggingface.co/blog/allenai/novelty-bench

First seen by Oriel on .

AI summary

Ai2’s study and benchmark find that LLM-based scientific novelty judgments are highly sensitive to prompts, models, and evaluation setups, especially for AI-generated research ideas.

  • The benchmark includes 154 high-novelty and 145 lower-novelty human ideas mined from ICLR 2026 reviews, plus 154 ideas from a simple LLM generator.
  • For GPT-5.4, a small prompt change reduced Human+Generated pairwise accuracy from 93.5% to 40.9% and reversed more than half of verdicts.
  • GPT-5.6 Sol with iterative arXiv search achieved 0.78 macro-F1, versus Claude Opus 4.6 at 0.89, while costing roughly 10× more per decision.
  • A Claude Opus 4.6 majority-vote baseline reached 0.854 macro-F1 for $2.59, outperforming Idea Novelty Checker and Sakana AI’s AI Scientist in accuracy and cost.
  • The released benchmark, data, code, and automated pipeline support testing judges across prompts, models, research communities, and future peer-review datasets.

#llm evaluation #scientific discovery #benchmarking #research novelty

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
Hugging Face Bloghttps://huggingface.co/blog/allenai/novelty-benchhttps://huggingface.co/blog/allenai/novelty-bench Read discussion