Oriel β€” AI & technology worth noticing, every day.

MathLLMs/MathVision

Dataset Β· https://huggingface.co/datasets/MathLLMs/MathVision

Measuring Multimodal Mathematical Reasoning with the MATH-Vision Dataset [πŸ’» Github] [🌐 Homepage] [πŸ“Š Main Leaderboard ] [πŸ“Š Open Source Leaderboard ] [🌿 Wild Leaderboard ] [πŸ” Visualization] [πŸ“– Paper] 🌿 NEW: MATH-Vision-Wild MATH-Vision-Wild is a photographic, real-world variant of MATH-Vision. The same testmini problems are physically captured on printed paper, iPads, laptops, and projectors under varying lighting and angles β€” the conditions VLMs actually… See the full description on the dataset page: https://huggingface.co/datasets/MathLLMs/MathVision.

First seen by Oriel on .

AI summary

MATH-Vision is a benchmark of 3,040 visual mathematics problems showing a substantial gap between multimodal models and humans, with MATH-Vision-Wild exposing further photographic robustness issues.

  • The dataset covers 16 mathematical disciplines and five difficulty levels, using high-quality problems sourced from real math competitions.
  • MATH-Vision-Wild re-captures problems on printed paper, iPads, laptops, and projectors under varying lighting and angles.
  • Among listed models, only o4-mini improved on photographed problems, reaching 57.2%; other models declined by 12.35% to 52.58%.
  • The dataset provides Hugging Face loading examples, leaderboards, visual examples, and ongoing corrections for reported question errors.
  • The benchmark evaluates multimodal mathematical reasoning across subjects and is intended for researchers comparing LMM capabilities and robustness.

#multimodal reasoning #ζ•°ε­¦ benchmarks #computer vision #large multimodal models

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days Β· Up to 120 recorded samples per source and metric, with reposts kept separate Β· First and latest retained

Likes

Total likes reported by the source.

  • Hugging Face Datasets: 182, latest observation ; 0 over observed period since

Downloads (30d)

Rolling 30-day downloads. A change in this value is not new downloads during the observed period.

  • Hugging Face Datasets: 10,064, latest observation ; +17 over observed period since

Aliases

SourceExternal IDURL
Hugging Face DatasetsMathLLMs/MathVisionhttps://huggingface.co/datasets/MathLLMs/MathVision Read discussion