VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents
Paper · https://arxiv.org/abs/2611.04894
Multimodal agents use web and image tools to answer questions beyond their internal knowledge. Some search tasks use images mainly to identify an initial entity, then rely on text retrieval. These tasks offer limited insight into how agents connect information across images, objects, and search steps. We therefore focus on visual context management. This involves organizing and using visual information and linking it to other findings during search. Visually intensive tasks require agents to combine object appearance, local details, or spatial relations with external information. We introduce VisSearch-Bench, with 204 human-written tasks across 7 main domains and 25 subdomains. It covers Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation, and Visual-Structure-Guided Search. Each task provides a reference answer, traceable evidence, and scoring rules. Accuracy measures complete task success, while Process Score measures the correctness and completeness of individual results and subquestion answers. Across 16 multimodal models, the best system fully solves only 26.47% of tasks, even with search tools. These results highlight the challenge of linking visual evidence with search results to produce complete answers.
First seen by Oriel on .
AI summary
VisSearch-Bench evaluates multimodal agents’ ability to manage visual context and connect evidence across images, objects, and search steps in 204 long-horizon tasks.
- The benchmark spans seven main domains and 25 subdomains, covering five visual search task types.
- Tasks require combining object appearance, local details, or spatial relations with information retrieved externally.
- Each task includes a reference answer, traceable evidence, and scoring rules for complete success and individual-result correctness.
- Across 16 multimodal models, the best system fully solved only 26.47% of tasks despite having search tools.
#multimodal agents #visual search #benchmarking
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 22, latest observation ; +5 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 5, latest observation ; +3 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2611.04894 | https://arxiv.org/abs/2611.04894 Read discussion |