Oriel — AI & technology worth noticing, every day.

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Paper · https://arxiv.org/abs/2610.26860

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.

First seen by Oriel on .

AI summary

OpenVLHarness is a multimodal reasoning harness that combines heterogeneous tools, evidence rendering, persistent multimodal memory, and code execution to improve visual agentic tool-use.

  • A capability layer exposes task-level tool interfaces and converts heterogeneous tool outputs into model-readable observations.
  • Persistent memory stores images, structured data, and web references as addressable artifacts throughout each query.
  • A coding interface executes generated programs over stored data, allowing computations that combine results from multiple tools.
  • Across 22 benchmarks and eight open and proprietary backbones, it improves average scores by 9.7-12.6 points over matched no-tool baselines.
  • With GPT-6 Luna at medium reasoning effort, it exceeds maximum-effort baselines by 7.7-8.5 points while using at most 37% of their model-token cost.

#multimodal models #visual agents #tool use #multimodal memory

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 121, latest observation ; +48 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 1, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.26860https://arxiv.org/abs/2610.26860 Read discussion