Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing
Quick Answer
Moir enhances cross-domain knowledge editing by allowing models like OLMo-2 and Qwen-3 to self-direct their preservation covariance, achieving 79.9% accuracy on GSM8K after 20,000 edits, compared to 10.9% with Wikipedia.
Quick Take
This method bypasses reliance on static external corpora, making it a practical solution for modern lacking publicly available pre-training data.
Key Points
- Moir estimates preservation covariance directly from the model's decoding distribution.
- It requires no external data, making it a drop-in solution for covariance-based editors.
- Qwen-3 achieved 79.9% GSM8K accuracy after 20,000 AlphaEdit batch edits.
- Moir addresses the asymmetric degradation of language models post-training.
- Aligning preservation distribution with operative distribution is crucial for effective editing.
DeepSignal Analysis
What happened
Moir introduces a method for cross-domain knowledge editing that allows models to self-direct their preservation covariance. This approach has shown significant improvements in accuracy, achieving 79.9% on the GSM8K benchmark after 20,000 edits, compared to just 10.9% using Wikipedia as a reference. The method does not rely on static external corpora, making it suitable for modern language models.
Key evidence
- Moir enables models like OLMo-2 and Qwen-3 to estimate preservation covariance directly from their decoding distribution, enhancing knowledge editing capabilities.
- After 20,000 AlphaEdit batch edits, Qwen-3 achieved 79.9% accuracy on GSM8K, significantly outperforming the 10.9% accuracy with Wikipedia.
- Moir serves as a drop-in component for covariance-based editors and does not require external data, addressing the lack of publicly available pre-training data for many modern LLMs.
Why it matters
The ability to allow models to self-direct their knowledge editing is crucial as it addresses the limitations of static external datasets, which often do not reflect the current knowledge landscape. This method could lead to more effective and efficient updates to language models, ensuring they remain relevant and accurate in a rapidly changing world. The significant improvement in accuracy on benchmarks like GSM8K suggests that this approach could redefine how knowledge editing is performed in AI systems.
Paper Resources
Source Excerpt
While language models remain frozen at their training state, the world evolves continuously. Knowledge editing has emerged as a key alternative to full retraining, but its deployment is bottlenecked by the erosion of core capabilities: mathematical and programmatic reasoning collapse while encyclopedic recall remains intact. We trace this asymmetric degradation to a distributional mismatch. Covariance-based editors preserve only the subspaces spanned by their reference corpus, but fail to captur
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.