Oriel — AI & technology worth noticing, every day.

XYBench: Helping with the problem behind the question

Web page · https://huggingface.co/blog/allenai/xybench

First seen by Oriel on .

AI summary

XYBench is an 8,115-question benchmark showing that leading AI models often answer users’ mistaken approaches instead of identifying misconceptions and prioritizing solutions to their underlying goals.

  • Questions come from Stack Overflow, Stack Exchange, manually curated examples, and WikiHow, covering programming, cooking, home repair, and other procedural activities.
  • It separately measures solutions to stated and intended requests, answer emphasis, and whether responses discourage the mistaken approach.
  • On Stack Overflow, models included stated-request solutions in 75–86% of responses but underlying-goal solutions in 59–71%, emphasizing goals in only 35–41%.
  • Making intent explicit raised Sonnet 4.6’s underlying-goal solutions from 59% to 67% but lowered misconception identification from 47% to 34%.
  • Although six models selected goal-focused answers 71–84% of the time, generated responses focused on underlying goals only 35–41%, revealing a recognition-production gap.

#llm evaluation #ai assistants #pragmatics #benchmarks

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
Hugging Face Bloghttps://huggingface.co/blog/allenai/xybenchhttps://huggingface.co/blog/allenai/xybench Read discussion