Model Collapse as Cultural Evolution
Quick Answer
The study reveals that model collapse in LLMs like LLaMA-2-7B and Mistral-7B can be explained through cultural evolution theory, showing a non-monotonic trajectory in compositionality during self-training.
Quick Take
The study reveals that model collapse in LLMs like LLaMA-2-7B and Mistral-7B can be explained through cultural evolution theory, showing a non-monotonic trajectory in compositionality during self-training. This phenomenon is evidenced by significant predictive results and suggests that task-grounded filtering is essential for maintaining model integrity over generations.
Key Points
- Model collapse characterized by degradation of LLMs trained on their own outputs.
- Five falsifiable predictions derived from cultural evolution theory were tested.
- Compositionality shows a non-monotonic trajectory during unfiltered self-training.
- Results confirmed with large effect sizes, indicating strong predictive power.
- LLM regularization gradients align closely with human behavioral data.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 23054v1 Announce Type: new Abstract: Model collapse, the progressive degradation of LLMs trained on their own outputs, has been characterized statistically but lacks a linguistic explanation for which structures degrade, in what order, and why. We show that iterated learning theory from cultural evolution fills this gap.
We derive five falsifiable predictions, distinguish those uniquely discriminative for the theory from confirmatory ones, and test them by self-training LLaMA-2-7B and Mistral-7B over 10 generations in English, German, and Turkish. The critical discriminative finding: compositionality follows a non-monotonic trajectory (initially rising, then falling) under unfiltered self-training.
This signature persists with maximally regular seed data (ruling out noise removal) and is sustained only by task-grounded filtering, not random filtering, providing the first LLM-scale evidence for the compression-communication tradeoff. All predictions are confirmed with large effect sizes (Hedges' $g > 1. 6$; $\mathrm{BF}_{10} > 100$), and LLM regularization gradients closely match human behavioral data ($R^2 = 0. 94$).
These results reframe model collapse as a cultural transmission phenomenon and yield concrete principles for self-training pipeline design.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.