Parameter-Efficient Fine-Tuning with Learnable Rank
Quick Take
Learnable Rank LoRA (LR-LoRA) introduces a dynamic rank adjustment during fine-tuning, outperforming fixed-rank methods across various benchmarks. This approach shows significant layer-wise rank variation, particularly in transformer models, achieving state-of-the-art results in language understanding and commonsense reasoning tasks.
Key Points
- LR-LoRA allows the optimizer to learn the rank for each adapter layer.
- Significant layer-wise rank variation observed in attention and MLP layers.
- Achieves state-of-the-art performance on language understanding benchmarks.
- Consistently outperforms strong parameter-efficient fine-tuning baselines.
- Demonstrates that learnable rank is a more effective inductive bias.
Article Excerpt
From source RSS / original summaryarXiv:2606. 04325v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bias by optimizing in a low-dimensional subspace. In this work, we question whether a fixed-rank constraint is the most effective inductive bias for parameter-efficient fine-tuning.
We introduce *Learnable Rank LoRA (LR-LoRA)*, a PEFT method in which the adapter rank is learned during the training process. Instead of prescribing a uniform rank for all adapter layers, LR-LoRA allows the optimizer to determine the appropriate rank for each layer. Using this approach, we find substantial layer-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences.
Across a range of language understanding and commonsense reasoning benchmarks, LR-LoRA achieves state-of-the-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed-rank adaptations.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.