XYBench: Helping with the problem behind the question
Web page · https://huggingface.co/blog/allenai/xybench
First seen by Oriel on .
AI summary
XYBench is an 8,115-question benchmark showing that leading AI models often answer users’ mistaken approaches instead of identifying misconceptions and prioritizing solutions to their underlying goals.
- Questions come from Stack Overflow, Stack Exchange, manually curated examples, and WikiHow, covering programming, cooking, home repair, and other procedural activities.
- It separately measures solutions to stated and intended requests, answer emphasis, and whether responses discourage the mistaken approach.
- On Stack Overflow, models included stated-request solutions in 75–86% of responses but underlying-goal solutions in 59–71%, emphasizing goals in only 35–41%.
- Making intent explicit raised Sonnet 4.6’s underlying-goal solutions from 59% to 67% but lowered misconception identification from 47% to 34%.
- Although six models selected goal-focused answers 71–84% of the time, generated responses focused on underlying goals only 35–41%, revealing a recognition-production gap.
#llm evaluation #ai assistants #pragmatics #benchmarks
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Blog | https://huggingface.co/blog/allenai/xybench | https://huggingface.co/blog/allenai/xybench Read discussion |