Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models
Paper · https://arxiv.org/abs/2611.09897
arXiv:2611.09897v1 Announce Type: new Abstract: Understanding the origins of the modality gap in vision language models remains a significant challenge. This work investigates whether differences in information density between modality specific tokens at the embedding layer contribute to this imbalance. We show that text tokens enter the model with much lower semantic density than visual tokens, driving early layers to focus on processing text representations before combining them with vision. Through mechanistic analysis, we demonstrate that replacing vision token positions with their semantically closest text token counterparts elicits increased attention to those positions in early layers. These findings provide a foundation for addressing the modality gap and provide insight to improve multimodal reasoning in VLM architectures.
First seen by Oriel on .
AI summary
The paper attributes part of the vision-language modality gap to lower semantic density in text tokens, causing early model layers to process text before integrating visual information.
- The study examines information-density differences between modality-specific tokens at the embedding layer.
- Mechanistic analysis shows early layers focus on processing text representations before combining them with vision.
- Replacing vision token positions with semantically closest text-token counterparts increases early-layer attention to those positions.
- The findings offer a basis for improving multimodal reasoning in VLM architectures.
#vision-language models #multimodal reasoning #mechanistic analysis #embeddings
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.09897v1 | https://arxiv.org/abs/2611.09897 Read discussion |