Oriel — AI & technology worth noticing, every day.

What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

Paper · https://arxiv.org/abs/2610.23987

Fine-grained visual understanding requires distinguishing subtle differences in local attributes and spatial relations, yet remains challenging for existing vision-language models. Unified multimodal models (UMMs) offer a new opportunity through generation pathways that preserve rich visual information complementary to understanding. However, existing generation-assisted understanding methods mainly rely on generative supervision or explicit generation and have not systematically explored direct use of native generation-side representations. To address this gap, we systematically analyze what generation-side representations offer for fine-grained understanding, finding that they are more sensitive to fine-grained semantic changes, capture task-relevant changes missed by understanding-side representations, and depend on correct spatial organization for this advantage. Guided by these findings, we design GAFU, a post-training framework that unlocks native generation-side information for fine-grained understanding. It compresses generation-side visual representations into compact \(G\) tokens that preserve fine-grained visual details, enabling task-conditioned retrieval by the understanding pathway. Experiments show that GAFU consistently outperforms SFT baselines trained on the same data across diverse visual understanding tasks and UMM architectures, with maximum relative improvements of 20.00\%, 17.94\%, and 8.62\% on VStar-Bench, OmniSpatial, and VSR-Bench, respectively. Overall, GAFU converts complementary information from the generation pathway into gains in fine-grained visual understanding, providing an effective path toward synergy between generation and understanding in UMMs.

First seen by Oriel on .

AI summary

The paper introduces GAFU, a post-training framework that uses generation-side representations to improve fine-grained visual understanding in unified multimodal models.

  • The authors find generation-side representations are more sensitive to fine-grained semantic changes and capture task-relevant information missed by understanding-side representations.
  • This advantage depends on correct spatial organization of visual information.
  • GAFU compresses generation-side visual representations into compact G tokens that the understanding pathway retrieves according to the task.
  • Across diverse tasks and model architectures, GAFU beats same-data SFT baselines, with maximum relative gains of 20.00% on VStar-Bench, 17.94% on OmniSpatial, and 8.62% on VSR-Bench.

#multimodal models #computer vision #visual understanding

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 40, latest observation ; +8 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 2, latest observation ; +1 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.23987https://arxiv.org/abs/2610.23987 Read discussion