Inducing Spatial Locality in Vision Transformers through the Training Protocol
Quick Take
Training protocols can enhance spatial locality in Vision Transformers without large-scale pretraining.
Key Points
- Modern protocol shows improved attention concentration in early layers.
- CutMix significantly influences spatial locality in Vision Transformers.
- No independent effect from AutoAugment or Label Smoothing.
📖 Reader Mode
~2 min readAbstract:We investigate whether the training protocol can induce spatial locality in the early layers of a Vision Transformer (ViT) trained from scratch, without large-scale pretraining. Keeping the architecture and optimization procedure fixed, we compare a Baseline protocol with a Modern protocol (AutoAugment/ColorJitter, CutMix, and Label Smoothing) on CIFAR-10, CIFAR-100, and Tiny-ImageNet, characterizing each attention head via Mean Attention Distance (MAD) and normalized entropy. Across all three datasets, the Modern protocol produces more local and more concentrated attention in early layers; on CIFAR-100, the minimum MAD drops from 0.316 (Baseline) to 0.008 (Modern). To identify the source of this effect, we conduct an ablation study on CIFAR-100 by adding or removing each component individually. The results identify CutMix as the determining component within our experiments: all conditions with CutMix exhibit MAD 0.024, while all conditions without CutMix remain at MAD 0.210. AutoAugment and Label Smoothing show no independent effect on locality. Taken together, these findings suggest that the pressure to classify from partial image regions, induced by CutMix, can promote the emergence of local attention in Vision Transformers.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2605.16390 [cs.CV] |
| (or arXiv:2605.16390v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16390 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Eduardo Santiago [view email]
[v1]
Mon, 11 May 2026 19:31:33 UTC (308 KB)
— Originally published at arxiv.org
More from arXiv cs.CV
See more →arXiv cs.CV·Jinhao Jing, Zheng Ma, Jinwei Liang, Qiannian Zhao, Shawn Chen, Jing Yang, Por Lip Yee, Prayag Tiwari, Jingjing Bai, Benyou Wang, Lewei Lu, Zhan Su
1d ago FeaturedOriginal
GeoSym127K: Scalable Symbolically-verifiable Synthesis for Multimodal Geometric Reasoning
AI Summary
GeoSym127K introduces a scalable neuro-symbolic framework for enhanced geometric reasoning in multimodal models.