Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions
Quick Answer
This study investigates the alignment between an LLM's next-token distribution and the empirical next-token distribution (ENTD) from its training data.
Quick Take
This study investigates the alignment between an LLM's next-token distribution and the empirical next-token distribution (ENTD) from its training data. Findings reveal that while many inputs show high agreement, significant discrepancies exist for certain sequences, prompting a call for more research into data-centric mechanistic interpretability.
Key Points
- LLM's output distribution aligns closely with ENTD for many inputs.
- Average agreement increases with model scale and training compute.
- Significant discrepancies exist for a long tail of input sequences.
- Study encourages exploration of data-centric mechanistic interpretability.
- Findings highlight the importance of understanding model behavior from training data.
Paper Resources
📖 Reader Mode
~2 min readAbstract:In this paper, we study the connection between an LLM's output distribution and the data used to train it. Specifically, we study the degree to which an LLM's next-token distribution agrees with the empirical next-token distribution (ENTD) given the context in the training data. The ENTD is an appealing target because it is the unrestricted global minimizer of the next-token cross entropy loss used for pretraining, as well as an easily interpretable function of the pretraining corpus. We find that for a significant fraction of inputs, the LLM's distribution agrees with the ENTD almost perfectly, and the average agreement increases with model scale and training compute. Nevertheless, there is a long tail of input sequences where the LLM and ENTD differ significantly, and we examine several possible sources of this discrepancy across the transformer architecture, training procedure, and finite-sample noise in the ENTD estimate itself. More broadly, we hope our findings will encourage more work on ``data-centric mechanistic interpretability,'' a complement to standard mechanistic interpretability that opens the black box of how model behaviors arise from the data, rather than how they are encoded in the learned weights.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.14306 [cs.AI] |
| (or arXiv:2607.14306v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.14306 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zachary Izzo [view email]
[v1]
Wed, 15 Jul 2026 19:11:54 UTC (13,141 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System
The MEDA system utilizes large language models and symbolic regression to autonomously discover ordinary differential equations for biological systems, achieving strong structural recovery and biologically plausible models. It outperforms existing methods by integrating domain knowledge and mechanistic constraints, demonstrating effective retrieval and extrapolation capabilities.