Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning
Quick Answer
FisherAdapTune introduces a dynamic parameter selection method for fine-tuning pretrained models, leveraging Fisher geometry to enhance performance in segmentation tasks.
Quick Take
FisherAdapTune introduces a dynamic parameter selection method for fine-tuning pretrained models, leveraging Fisher geometry to enhance performance in segmentation tasks. The framework shows improved in-distribution performance and zero-shot transfer, demonstrating the effectiveness of tracking Fisher structural drift for efficient adaptation.
Key Points
- FisherAdapTune tracks Fisher geometry to select parameter groups dynamically.
- The method reduces generalization error by freezing stabilized parameter groups.
- Evaluation on segmentation tasks shows significant performance improvements.
- Results indicate better zero-shot transfer capabilities across multiple settings.
- Code for FisherAdapTune is publicly available for further research.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 10196v1 Announce Type: new Abstract: Parameter-efficient fine-tuning (PEFT) aims to adapt pretrained models with a small trainable parameter subset, however, most existing methods choose this subset from fixed architectural heuristics rather than using dynamic, task-aware criteria. We introduce \textbf{FisherAdapTune}, a Fisher-guided Adaptive Fine-Tuning framework that progressively selects parameter groups by tracking the temporal drift of their Fisher geometry.
Starting from a PAC-Bayesian view of fine-tuning, we decompose the generalization error bound into Fisher-weighted update costs and show that parameter groups whose curvature contribution has stabilized can be frozen to reduce the error bound without interrupting the remaining adaptation dynamics. FisherAdapTune formulates this criterion with a scale-invariant Jensen-Shannon distance between consecutive Fisher distributions, yielding an adaptive active parameter set.
We evaluate our approach on a downstream segmentation task, and results show FisherAdapTune improves the in-distribution performance and zero-shot transfer in multiple settings, validating that Fisher structural drift is a useful signal for efficient, task-aware adaptation. We release our \href{https://github. com/AtlasAnalyticsLab/FisherAdapTune}{code} publicly to enable further application of our proposed approach.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.