Information-Theoretic Limits of Reliability and Scaling in Language Models
Quick Answer
This paper reveals that large language models (LLMs) face an information-theoretic reliability ceiling based on output uncertainty and task ambiguity.
Quick Take
This paper reveals that large language models (LLMs) face an information-theoretic reliability ceiling based on output uncertainty and task ambiguity. The authors derive a scaling law that links model performance to the limitations of training data and model capacity, providing insights into when scaling improves reliability and unifying various phenomena in generative models.
Key Points
- Reliability ceiling for generative tasks is determined by observable context and task ambiguity.
- Autoregressive generation degrades reliability based on inter-token correlations.
- A new scaling law links LLM performance to training data and model capacity.
- The framework explains benefits of retrieval-augmentation and catastrophic forgetting.
- Offers a unified theory of performance limits in generative language models.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language models (LLMs) are evaluated as though perfect reliability is achievable for any task given sufficient scale. We show this assumption is information-theoretically unjustified. Every generative task has a reliability ceiling that no model can exceed, determined by how much output uncertainty is resolvable from observable context. The gap decomposes into a resolvable component closable with additional context and a subjective component inherent to task ambiguity. Autoregressive generation further degrades this ceiling at a rate governed by the task's dependency kernel, which quantifies inter-token correlations in the output. From these two primitives, we derive a first-principles scaling law where LLM performance is bottlenecked by the scarcer resource: training data or model capacity. This law recovers the Chinchilla scaling law as a special case and provides a structural account of when scaling improves reliability. Beyond scaling, our framework unifies diverse practical phenomena, such as the benefits of retrieval-augmentation and the spectral mechanics of catastrophic forgetting. Our work formalizes the resource-complexity tradeoffs that govern model performance across domains, offering a unified theory of performance limits in generative language models.
| Comments: | 29 pages, 2 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Theory (cs.IT) |
| Cite as: | arXiv:2607.14112 [cs.CL] |
| (or arXiv:2607.14112v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.14112 arXiv-issued DOI via DataCite |
Submission history
From: Subhabrata Majumdar [view email]
[v1]
Fri, 8 May 2026 06:06:18 UTC (82 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.