Retrieval-Augmented Linguistic Calibration
17h ago
Quick Take
RALC enhances linguistic confidence calibration using a distributional framework and retrieval-augmented rewriting.
Key Points
- Models linguistic confidence as a distribution over perceived probabilities.
- Introduces Faithfulness Divergence to evaluate audience belief surprises.
- Achieves up to 66% improvement in faithfulness across QA benchmarks.
📖 Reader Mode
~2 min readAbstract:Linguistic cues such as "I believe" and "probably" offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic confidence expressions remains underexplored. In particular, co-occurring linguistic cues, contextual variation, and subjective audience interpretation pose unique challenges. We therefore model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing interpretation variability that scalar representations discard. Within this distributional framework, we introduce faithfulness as a complementary evaluation dimension and present Faithfulness Divergence (FD), an information-theoretic metric quantifying the surprise induced in audience beliefs upon truth revelation. Building on these foundations, we present Retrieval-Augmented Linguistic Calibration (RALC), a lightweight post-hoc pipeline that propagates calibrated confidence signals back into natural language via retrieval-augmented rewriting. Across three QA benchmarks and five LLM families, RALC improves in-domain faithfulness and calibration up to 66% and 58%, respectively, outperforming black-box and grey-box calibration baselines.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.19344 [cs.CL] |
| (or arXiv:2605.19344v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19344 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yi-Fan Yeh [view email]
[v1]
Tue, 19 May 2026 04:31:38 UTC (610 KB)
— Originally published at arxiv.org
More from arXiv cs.CL
See more →arXiv cs.CL·Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, Arman Cohan
17h ago FeaturedOriginal
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
AI Summary
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.