Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning
Quick Answer
FisherAdapTune introduces a dynamic parameter selection method for fine-tuning pretrained models, leveraging Fisher geometry to enhance performance in segmentation tasks.
Quick Take
The framework shows improved in-distribution performance and zero-shot transfer, demonstrating the effectiveness of tracking Fisher structural drift for efficient adaptation.
Key Points
- FisherAdapTune tracks Fisher geometry to select parameter groups dynamically.
- The method reduces generalization error by freezing stabilized parameter groups.
- Evaluation on segmentation tasks shows significant performance improvements.
- Results indicate better zero-shot transfer capabilities across multiple settings.
- Code for FisherAdapTune is publicly available for further research.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 10196v1 Announce Type: new Abstract: Parameter-efficient fine-tuning (PEFT) aims to adapt pretrained models with a small trainable parameter subset, however, most existing methods choose this subset from fixed architectural heuristics rather than using dynamic, task-aware criteria. We introduce \textbf{FisherAdapTune}, a Fisher-guided Adaptive Fine-Tuning framework that progressively selects parameter groups by tracking the temporal drift of their Fisher geometry.
Starting from a PAC-Bayesian view of fine-tuning, we decompose the generalization error bound into Fisher-weighted update costs and show that parameter groups whose curvature contribution has stabilized can be frozen to reduce the error bound without interrupting the remaining adaptation dynamics. FisherAdapTune formulates this criterion with a scale-invariant Jensen-Shannon distance between consecutive Fisher distributions, yielding an adaptive active parameter set.
We evaluate our approach on a downstream segmentation task, and results show FisherAdapTune improves the in-distribution performance and zero-shot transfer in multiple settings, validating that Fisher structural drift is a useful signal for efficient, task-aware adaptation. We release our \href{https://github. com/AtlasAnalyticsLab/FisherAdapTune}{code} publicly to enable further application of our proposed approach.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.