When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking
Quick Answer
The study introduces Training-Free Gated Reranking, which leverages model uncertainty to determine reranking necessity, achieving 15%-80% cost reduction and up to 2% performance improvement across 8 LLMs on 7 NLU datasets.
Quick Take
The study introduces Training-Free Gated Reranking, which leverages model uncertainty to determine reranking necessity, achieving 15%-80% cost reduction and up to 2% performance improvement across 8 LLMs on 7 NLU datasets. This challenges the assumption that reranking always enhances performance, emphasizing its effectiveness for high-uncertainty instances.
Key Points
- Training-Free Gated Reranking reduces computational costs by 15%-80%.
- Performance improvements reach up to 2% across various datasets.
- The approach is validated on 8 large language models.
- Reranking is most beneficial for high-uncertainty instances.
- Challenges the belief that reranking always enhances performance.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2606. 31087v1 Announce Type: new Abstract: Few-shot selection typically assumes that reranking retrieved examples always improves performance. We challenge this view by identifying that the expensive reranking step can in fact degrade performance. Instead, we propose \emph{Training-Free Gated Reranking}, which decides whether to rerank the few-shot examples based on the model's uncertainty.
Extensive experiments across 8 LLMs, covering 7 NLU datasets and 9 MT domain-language combinations, demonstrate that our approach reduces computational costs by 15\%-80\% while improving average performance by up to 2\%. These findings indicate that higher computational cost does not guarantee better performance, and that reranking is most beneficial when targeted at high-uncertainty instances.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.