Oriel — AI & technology worth noticing, every day.

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Paper · https://arxiv.org/abs/2610.28444

Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.

First seen by Oriel on .

AI summary

EvaOmni is an evidence-grounded benchmark for evaluating whether audio-visual language models support their answers with correctly localized evidence, not just answer correctly.

  • The benchmark contains 392 videos and 1,027 human-annotated question-answer pairs spanning 15 capabilities and three reasoning levels.
  • Models must return both an answer and supporting audio and visual evidence intervals; Coverage Grounded Score measures evidence coverage, while Bottleneck Grounded Score penalizes poor temporal localization.
  • Annotations include expert-labeled evidence, video-based distractors, human review, and model filtering for textual and unimodal shortcuts.
  • Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash led overall with 79.4% accuracy, 70.8% C.G., and 35.1% B.G.
  • On Level 3 tasks, Gemini3.7-Flash achieved 79.1% accuracy but only 12.3% B.G., indicating weak evidence localization despite correct answers.

#audio-visual reasoning #omni llms #benchmarking

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 18, latest observation ; +1 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 2, latest observation ; +1 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.28444https://arxiv.org/abs/2610.28444 Read discussion