Mind the Heads: Topological Representation Alignment for Multimodal LLMs
Quick Answer
The proposed Head-Wise Representation Alignment (HeRA) method enhances Multimodal Large Language Models (MLLMs) by aligning individual attention heads, improving performance on vision-centric tasks across 18 benchmarks.
Quick Take
HeRA effectively reduces visual hallucinations by focusing on the least aligned heads, demonstrating significant gains in model robustness.
Key Points
- HeRA aligns individual attention heads for improved MLLM performance.
- Extensive evaluations show consistent gains across 18 vision-centric benchmarks.
- The method reduces visual hallucinations by curbing reliance on linguistic priors.
- Aligning the least aligned heads yields the largest performance improvements.
- Code for HeRA is publicly available for further research.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multimedia (cs.MM) |
| Cite as: | arXiv:2606.23885 [cs.CV] |
| (or arXiv:2606.23885v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.23885 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Davide Caffagni [view email]
[v1]
Mon, 22 Jun 2026 19:30:30 UTC (2,104 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.