LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment
Quick Answer
LoCA introduces a two-stage method for tuning LLMs, achieving up to 29% lower GPU peak usage and 52% lower CPU memory compared to LoRA.
Quick Take
Evaluated on Qwen2.5 models, it outperformed LoRA in 16 of 25 benchmarks, enabling efficient forward-only tuning post-calibration.
Key Points
- LoCA replaces repeated backpropagation with a one-time calibration process.
- Achieves lower evaluation cross-entropy than LoRA in 16 out of 25 benchmarks.
- Full-run GPU peak usage is 26-29% lower than LoRA's.
- CPU steady-state memory is reduced by 36-52% after calibration.
- Enables forward-only tuning when repeated backpropagation is impractical.
DeepSignal Analysis
What happened
LoCA is a new two-stage method for tuning large language models (LLMs) that reduces the need for repeated backpropagation. It achieves significant reductions in GPU peak usage and CPU memory compared to LoRA, particularly when evaluated on Qwen2.5 models across various benchmarks.
Key evidence
- LoCA reduces GPU peak usage by 26-29% and CPU steady-state memory by 36-52% compared to LoRA during model tuning.
- In evaluations on Qwen2.5 models, LoCA outperformed LoRA in 16 out of 25 benchmarks, demonstrating its effectiveness in various tasks.
- The method allows for forward-only tuning after a one-time calibration, making it suitable for scenarios where repeated backpropagation is impractical.
Why it matters
The introduction of LoCA could represent a significant advancement in the efficiency of tuning large language models. By minimizing the reliance on backpropagation, it may enable broader accessibility and usability of LLMs in resource-constrained environments. This could lead to faster deployment and lower operational costs for organizations leveraging AI technologies.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAbstract:Parameter-efficient post-training reduces the number of trainable parameters, but still requires repeated end-to-end backpropagation through the frozen backbone. Every adaptation step therefore needs backward-capable hardware and must store or recompute activations. We ask whether this repeated backward chain can be replaced by a one-time calibration. We introduce Local Credit Assignment (LoCA), a two-stage method for small-shift adaptation. One probe backward pass fits a low-rank map at each transformer block from the final prediction error to a local hidden-state correction. LoCA then reuses these maps to form blockwise regression targets from forward activations and fits low-rank adapters with closed-form ridge solves. No further backbone backward pass is required. We evaluate LoCA on five discriminative benchmarks with Qwen2.5 models from 0.5B to 14B. In 16 of 25 reported task--scale comparisons, LoCA yields lower evaluation cross-entropy than the corresponding LoRA run. Its measured full-run GPU peak, including calibration, is 26--29\% lower than LoRA's. After calibration, its CPU steady-state memory is 36--52\% lower and its per-pass time is 43--48\% lower. A shared scale-normalized candidate set is reused across all tested Qwen2.5 sizes and on SmolLM2-1.7B. LoCA thus amortizes global credit assignment into one calibration and enables later forward-only tuning when repeated backpropagation is impractical. The code associated with this paper is available \href{this https URL}{here}.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.03020 [cs.AI] |
| (or arXiv:2608.03020v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.03020 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Linhan Xia [view email]
[v1]
Tue, 4 Aug 2026 02:06:43 UTC (269 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.
