A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers
Quick Answer
This study investigates adversarial fine-tuning of Vision Transformers (ViTs) to enhance robustness against image perturbations.
Quick Take
While fine-tuning improves performance on familiar corruptions, it fails to generalize to unseen types. The analysis reveals changes in attention mechanisms but no fundamental shifts in sparse representations.
Key Points
- Adversarial fine-tuning improves ViT performance on familiar image corruptions.
- Improvements do not transfer to unseen types of image perturbations.
- Changes in attention mechanisms were observed during the analysis.
- No fundamental changes in sparse representations of ViTs were found.
- Study emphasizes the need for robust models in high-risk applications.
Paper Resources
Source Excerpt
arXiv:2606. 07593v1 Announce Type: new Abstract: The widespread use of image classification models in high-risk, real-world situations necessitates making these models robust to slight disturbances or perturbations, such as blurring or sharpening, in the input images. While vision transformers (ViTs) play an integral role in many modern-day multi-modal models like Vision-Language-Models () and Vision-Language-Action (VLA) models, they have received a lack of attention in the setting of robustness. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.