Memorization Dynamics of Fill-in-the-Middle Pretraining
Quick Answer
The study examines the memorization dynamics of Fill-in-the-Middle (FIM) pretraining in Llama 3.2 models, revealing that FIM enhances recovery of short spans while LTR excels in long exact continuations.
Quick Take
Verbatim recall under FIM grows linearly with repetitions, emphasizing the importance of prefix context over suffix context.
Key Points
- FIM pretraining enhances infilling ability in Llama 3.2 models.
- FIM recovers short spans more effectively than standard left-to-right (LTR) training.
- Verbatim recall under FIM training increases linearly with repetitions.
- Prefix context is crucial for verbatim recall in FIM training.
- Evaluating only one span length can overlook key memorization nuances.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 22981v1 Announce Type: new Abstract: Fill-in-the-middle (FIM) is a pretraining objective widely used to equip causal language models with infilling ability, yet its effect on verbatim memorization remains underexplored. We study the memorization dynamics of FIM in a controlled setting by pretraining matched Llama 3. 2 models with FIM and standard left-to-right (LTR) objectives on a FineWeb-Gutenberg corpus containing repeated Gutenberg excerpts.
With prefix-based probes, FIM more often recovers short or partially matching spans, while LTR more often assigns high confidence to long exact continuations. We observe that verbatim extraction under FIM-training grows approximately linearly with repetitions over the tested range. Evaluating native FIM-format probes reveals that suffix context is not sufficient: verbatim recall under FIM-training remains strongly anchored in prefix context.
Our results also show that evaluating only one span length or probing format can miss important nuances in memorization behavior.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.