Linguistic Productivity in Large Language Models: Models Coerce, but do not Preempt
Quick Take
This study explores how large language models (LLMs) like GPT-3 exhibit linguistic productivity influenced by entrenchment and preemption. It finds that while LLMs can generate novel constructions through coercion, they do not effectively utilize negative evidence to avoid overgeneralization of unseen patterns.
Key Points
- Linguistic productivity in LLMs is shaped by frequency signals of entrenchment and preemption.
- Larger models demonstrate constructional productivity with nonce words in coercive contexts.
- Even the largest LLMs do not apply negative evidence to novel language structures.
- Statistical preemption fails to prevent overgeneralization of semantically plausible patterns.
Article Content
From source RSS / original summaryarXiv:2606. 02953v1 Announce Type: new Abstract: Usage-based theories of grammars posit that creative productivity of the structures of language is both bolstered and constrained by two distinct frequency signals: entrenchment, stemming from high frequency usage, and preemption, stemming from having never observed a particular linguistic structure in a context where one might expect that structure to appear.
Large Language Models are also usage-based, in the sense that the structures of language are learned through exposure to vast amounts of text. Here, we test whether or not the opposing statistical forces of entrenchment and preemption also encourage and constrain linguistic productivity in LLMs.
We demonstrate across model architectures that larger models recognize and can reproduce with nonce words constructional productivity (entrenchment) in cases of coercion, wherein the broader constructional context coerces an atypical interpretation of a lexical item.
However, we also show that even the largest models do not extend negative evidence to novel language, and statistical preemption does not enable models to avoid overgeneralization of patterns that are semantically felicitous, but never observed in data.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.