Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions
Quick Answer
This study investigates the alignment between an LLM's next-token distribution and the empirical next-token distribution (ENTD) from its training data.
Quick Take
Findings reveal that while many inputs show high agreement, significant discrepancies exist for certain sequences, prompting a call for more research into data-centric mechanistic interpretability.
Key Points
- 's output distribution aligns closely with ENTD for many inputs.
- Average agreement increases with model scale and training compute.
- Significant discrepancies exist for a long tail of input sequences.
- Study encourages exploration of data-centric mechanistic interpretability.
- Findings highlight the importance of understanding model behavior from training data.
Paper Resources
📖 Reader Mode
~2 min readAbstract:In this paper, we study the connection between an LLM's output distribution and the data used to train it. Specifically, we study the degree to which an LLM's next-token distribution agrees with the empirical next-token distribution (ENTD) given the context in the training data. The ENTD is an appealing target because it is the unrestricted global minimizer of the next-token cross entropy loss used for pretraining, as well as an easily interpretable function of the pretraining corpus. We find that for a significant fraction of inputs, the LLM's distribution agrees with the ENTD almost perfectly, and the average agreement increases with model scale and training compute. Nevertheless, there is a long tail of input sequences where the LLM and ENTD differ significantly, and we examine several possible sources of this discrepancy across the transformer architecture, training procedure, and finite-sample noise in the ENTD estimate itself. More broadly, we hope our findings will encourage more work on ``data-centric mechanistic interpretability,'' a complement to standard mechanistic interpretability that opens the black box of how model behaviors arise from the data, rather than how they are encoded in the learned weights.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.14306 [cs.AI] |
| (or arXiv:2607.14306v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.14306 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zachary Izzo [view email]
[v1]
Wed, 15 Jul 2026 19:11:54 UTC (13,141 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.