Oriel — AI & technology worth noticing, every day.

CulturalCrosswords: A Multilingual Benchmark comprised of Answer Constraints and Grid Completion

Paper · https://arxiv.org/abs/2611.10044

arXiv:2611.10044v1 Announce Type: new Abstract: When several answers fit a riddle, exact-match scoring can penalise a valid response. CulturalCrosswords tests how answer constraints affect large language models (LLMs) solving cultural riddles, whose solutions draw on knowledge specific to a community. We constructed 53 crosswords from crowdsourced riddles in 42 languages and evaluated 11 LLM configurations on five versions of each clue set: plain batches, batches with lengths, batches with lengths and partial-answer hints, and crosswords requesting an answer list or a completed grid. On 962 clues with alternative answers, lengths improve agreement with the reference by 22.8 percentage points, compared with 8.4 points when accepted alternatives also receive credit. Part of the gain therefore appears to come from selecting a required answer form. Crossword input increases the share of all crossings with compatible answers by 9.7 points; requiring a completed grid reveals large differences between models. The benchmark distinguishes agreement with a reference from the ability to produce answers that fit together and occupy the correct cells.

First seen by Oriel on .

No saved summary available.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.10044v1https://arxiv.org/abs/2611.10044 Read discussion