When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning
Quick Answer
The paper introduces Tail-Aware Credit Calibration (TACO) to address Positive-Credit Contamination in reinforcement learning for large language models (LLMs).
Quick Take
TACO improves training stability and performance across three and eight benchmarks by calibrating credit assignment, effectively distinguishing between useful rare patterns and noise. Experimental results show TACO consistently outperforms -style baselines.
Key Points
- TACO calibrates credit assignment to mitigate flawed reasoning in LLMs.
- The method computes a tail-risk score to assess token reliability.
- TACO enhances training stability and supports long-horizon RL performance.
- Experimental results show consistent improvements over GRPO-style baselines.
- Source code for TACO is publicly available.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token's risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration. TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.07976 [cs.CL] |
| (or arXiv:2607.07976v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07976 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zicheng Xu [view email]
[v1]
Wed, 8 Jul 2026 23:00:54 UTC (1,901 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.