In-Context Collapse in Vision-Language Models and How to Mitigate it?
Quick Answer
This paper shows that Many-shot in-context learning in vision-language models can lead to in-context collapse, causing accuracy drops below chance in tasks like classification and VQA.
Quick Take
This phenomenon is observed across various models, including Claude Sonnet 4.5, and can be mitigated by using a lightweight adapter that enhances vision-language integration, improving performance significantly.
Key Points
- In-context collapse leads to accuracy drops below chance in some .
- Observed in models ranging from 0.5B to 11B parameters, including Claude Sonnet 4.5.
- Robustness to demonstrations and learning novel rules are dissociable capabilities.
- A lightweight adapter can restore learning accuracy from 0.39 to 0.91 at 16 shots.
- Proposed CircA transfers collapse-resistance to unseen tasks, improving performance.
DeepSignal Analysis
What happened
The study reveals that many-shot in-context learning in vision-language models can lead to in-context collapse, resulting in significant accuracy drops in tasks like classification and VQA. This collapse occurs in various models, including Claude Sonnet 4.5. A lightweight adapter can mitigate this issue by enhancing vision-language integration.
Key evidence
- In-context collapse can cause accuracy drops below chance in tasks such as classification and VQA, affecting a subset of vision-language models.
- The collapse is observed across a range of models, including those with parameters from 0.5B to 11B, and is graded in severity.
- A lightweight adapter applied to the vision-language integration pathway can restore learning capabilities, improving accuracy from 0.39 to 0.91 at 16 shots.
Why it matters
Understanding in-context collapse is crucial for improving the reliability of vision-language models, especially as they are increasingly used in real-world applications. The ability to mitigate this collapse through lightweight interventions could enhance model performance and broaden their applicability across various tasks. This insight may lead to more robust AI systems capable of better understanding and integrating visual and textual information.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied. We show the opposite: as demonstrations accumulate, a subset of VLMs undergo an \emph{in-context collapse}, a sharp, sometimes catastrophic accuracy drop spanning synthetic classification, natural-image classification, and VQA benchmarks, in some models falling below chance while outputs remain well-formed. Across an open VLM panel ($0.5$B--$11$B) and a frontier model (Claude Sonnet 4.5), the collapse is graded. Two capabilities turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context, their combinations yield three reproducible regimes. A parameter-matched lesion-and-rescue causally localizes the collapse to the vision-language integration pathway: an adapter on the connector and early/mid layers restores genuine learning (remap accuracy $0.39!\rightarrow!0.91$ at 16 shots), while an equal-capacity adapter on the late readout does not. We propose \textsc{CircA}, whose core is a one-time integration vaccine: trained once on one synthetic task, it transfers collapse-resistance to unseen task families (chance$\rightarrow$$0.71$/$0.60$ on CIFAR/Fashion). The layers best for in-context integration are not the layers best for weight-based consolidation, the late readout achieves higher accuracy and less forgetting at fewer parameters. The collapse is an integration failure at the vision--language interface, correctable by a lightweight, transferable intervention.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.02830 [cs.CV] |
| (or arXiv:2608.02830v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2608.02830 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mohammad Rostami [view email]
[v1]
Mon, 3 Aug 2026 19:44:45 UTC (1,453 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.