When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning
Quick Answer
The paper introduces Tail-Aware Credit Calibration (TACO) to address Positive-Credit Contamination in reinforcement learning for large language models (LLMs).
Quick Take
The paper introduces Tail-Aware Credit Calibration (TACO) to address Positive-Credit Contamination in reinforcement learning for large language models (LLMs). TACO improves training stability and performance across three LLMs and eight benchmarks by calibrating credit assignment, effectively distinguishing between useful rare patterns and noise. Experimental results show TACO consistently outperforms GRPO-style baselines.
Key Points
- TACO calibrates credit assignment to mitigate flawed reasoning in LLMs.
- The method computes a tail-risk score to assess token reliability.
- TACO enhances training stability and supports long-horizon RL performance.
- Experimental results show consistent improvements over GRPO-style baselines.
- Source code for TACO is publicly available.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token's risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration. TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.07976 [cs.CL] |
| (or arXiv:2607.07976v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07976 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zicheng Xu [view email]
[v1]
Wed, 8 Jul 2026 23:00:54 UTC (1,901 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.