Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum
8h ago
Quick Take
The study links data scaling laws to a latent predictive contribution spectrum rather than just token-frequency tails.
Key Points
- Investigates predictive contribution spectrum in data scaling.
- Empirical correlation found across 12 real corpora.
- Effective truncation rank matches observed excess loss.
📖 Reader Mode
~2 min readAbstract:We investigate the hypothesis that real-data scaling laws are governed by progressive coverage of a latent predictive contribution spectrum rather than by token-frequency tails alone. We work with a suffix-automaton representation of text corpora and define a data-intrinsic global-KL predictive contribution spectrum, in which each state contributes according to its empirical mass times its KL deviation from a global next-token baseline. Across 12 real corpora, the tail slope of this spectrum is already strongly correlated with the empirical data-scaling exponent of a fixed small GPT learner. We then go beyond slope correlation and define, for each training size N, an effective truncation rank K(N) by matching the observed excess loss to the residual tail mass of the prepared 1000k global-KL spectrum. Empirically, log K is close to linear in log N, with pooled R^2 about 0.96 for the raw spectrum and R^2 about 0.90 for the smoothed spectrum. These findings provide strong empirical support for a simple mechanism picture: training scale advances an effective frontier through a predictive state spectrum, and the residual tail mass of that spectrum tracks the remaining excess loss.
| Comments: | 8 pages,6 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.20196 [cs.CL] |
| (or arXiv:2605.20196v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.20196 arXiv-issued DOI via DataCite |
Submission history
From: HongXi Li [view email]
[v1]
Sun, 5 Apr 2026 10:47:24 UTC (1,382 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →arXiv cs.CL·Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, Arman Cohan
1d ago FeaturedOriginal
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
AI Summary
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.