Oriel — AI & technology worth noticing, every day.

Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models

Paper · https://arxiv.org/abs/2611.09897

arXiv:2611.09897v1 Announce Type: new Abstract: Understanding the origins of the modality gap in vision language models remains a significant challenge. This work investigates whether differences in information density between modality specific tokens at the embedding layer contribute to this imbalance. We show that text tokens enter the model with much lower semantic density than visual tokens, driving early layers to focus on processing text representations before combining them with vision. Through mechanistic analysis, we demonstrate that replacing vision token positions with their semantically closest text token counterparts elicits increased attention to those positions in early layers. These findings provide a foundation for addressing the modality gap and provide insight to improve multimodal reasoning in VLM architectures.

First seen by Oriel on .

AI summary

The paper attributes part of the vision-language modality gap to lower semantic density in text tokens, causing early model layers to process text before integrating visual information.

  • The study examines information-density differences between modality-specific tokens at the embedding layer.
  • Mechanistic analysis shows early layers focus on processing text representations before combining them with vision.
  • Replacing vision token positions with semantically closest text-token counterparts increases early-layer attention to those positions.
  • The findings offer a basis for improving multimodal reasoning in VLM architectures.

#vision-language models #multimodal reasoning #mechanistic analysis #embeddings

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.09897v1https://arxiv.org/abs/2611.09897 Read discussion