Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis
Quick Take
Language models underpredict human reading difficulty due to differences in parse interpretation capabilities.
Key Points
- Surprisal theory links word predictability to processing difficulty.
- LMs can consider more sentence interpretations simultaneously.
- Reducing active parses increases predicted garden path effects.
📖 Reader Mode
~2 min readAbstract:Surprisal theory posits that the processing difficulty of a word is determined by its predictability in context, offering a potential link between human sentence processing and next-word predictions from language models. While language model (LM) surprisals successfully predict reading times in naturalistic text, they systematically underpredict the magnitude of difficulty observed in controlled studies of syntactic ambiguity, particularly in garden path sentences. This mismatch might arise from differences in the computational constraints between humans and LMs. Here we test one such hypothesis, specifically, that LMs may be able to simultaneously consider a greater number of distinct sentence interpretations at once, compared to humans. Using Recurrent Neural Network Grammars (RNNGs) with word-synchronous beam search, we systematically vary the number of simultaneous parses used to compute word surprisal, and then use these surprisals to predict human reading times. Reducing the number of simultaneous active parses indeed increases the magnitude of predicted garden path effects, but not nearly enough to capture the full magnitude of the effects in humans. This suggests that differences in the number of simultaneous parses available to LMs and humans cannot reconcile LM-based surprisal with human sentence processing.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.15440 [cs.CL] |
| (or arXiv:2605.15440v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15440 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: William Timkey [view email]
[v1]
Thu, 14 May 2026 21:44:26 UTC (618 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →arXiv cs.CL·Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, Arman Cohan
2d ago FeaturedOriginal
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
AI Summary
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.
