Do Language Models Know What Not to Say? Causal Evidence for Statistical Preemption in LLMs
Quick Take
This study reveals that large language models (LLMs) like GPT-3 acquire knowledge of unacceptable language through statistical preemption, showing a strong correlation (r = 0.79) with human acceptability judgments across 120 verb-construction pairings. The findings suggest that LLMs learn negative linguistic knowledge via distributional competition, challenging the entrenched hypothesis.
Key Points
- LLMs show strong correlation (r = 0.79) with human judgments on language acceptability.
- Preemption sensitivity scales with model size, indicating increased learning capacity.
- Competing-form frequency, not overall verb frequency, drives LLM surprisal patterns.
- Fine-tuning interventions successfully shift preemption behavior in predicted directions.
- Study provides evidence for distributional competition as a learning mechanism in LLMs.
Article Content
From source RSS / original summaryarXiv:2605. 23039v1 Announce Type: new Abstract: How do learners acquire knowledge of what is unacceptable without negative evidence? Construction Grammar proposes statistical preemption: exposure to a conventional form (e. g. , "donated the books to the library") preempts structurally possible but unattested alternatives ("*donated the library the books").
We present a computational study that, for the first time, directly dissociates statistical preemption from the competing entrenchment hypothesis in large language models within a single converging design. Across four experiments spanning 120 English verb-construction pairings (dative, causative, locative), we show that (1) LLM surprisal patterns correlate strongly with human acceptability judgments ($r = 0.
79$), validated against three independent behavioral datasets; (2) these patterns are driven by competing-form frequency rather than overall verb frequency, confirmed by non-circular partial correlations; (3) preemption sensitivity scales as a power law with model size; and (4) a controlled fine-tuning intervention causally demonstrates that manipulating competing-form frequencies shifts preemption behavior in the predicted direction, with reverse-direction controls ruling out frequency-sensitivity confounds.
These results provide converging evidence that neural language models acquire negative linguistic knowledge through distributional competition, the core mechanism posited by Construction Grammar.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.