Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing
Quick Answer
Moir enhances cross-domain knowledge editing by allowing models like OLMo-2 and Qwen-3 to self-direct their preservation covariance, achieving 79.9% accuracy on GSM8K after 20,000 edits, compared to 10.9% with Wikipedia.
Quick Take
This method bypasses reliance on static external corpora, making it a practical solution for modern lacking publicly available pre-training data.
Key Points
- Moir estimates preservation covariance directly from the model's decoding distribution.
- It requires no external data, making it a drop-in solution for covariance-based editors.
- Qwen-3 achieved 79.9% GSM8K accuracy after 20,000 AlphaEdit batch edits.
- Moir addresses the asymmetric degradation of language models post-training.
- Aligning preservation distribution with operative distribution is crucial for effective editing.
DeepSignal Analysis
What happened
Moir introduces a method for cross-domain knowledge editing that allows models to self-direct their preservation covariance. This approach has shown significant improvements in accuracy, achieving 79.9% on the GSM8K benchmark after 20,000 edits, compared to just 10.9% using Wikipedia as a reference. The method does not rely on static external corpora, making it suitable for modern language models.
Key evidence
- Moir enables models like OLMo-2 and Qwen-3 to estimate preservation covariance directly from their decoding distribution, enhancing knowledge editing capabilities.
- After 20,000 AlphaEdit batch edits, Qwen-3 achieved 79.9% accuracy on GSM8K, significantly outperforming the 10.9% accuracy with Wikipedia.
- Moir serves as a drop-in component for covariance-based editors and does not require external data, addressing the lack of publicly available pre-training data for many modern LLMs.
Why it matters
The ability to allow models to self-direct their knowledge editing is crucial as it addresses the limitations of static external datasets, which often do not reflect the current knowledge landscape. This method could lead to more effective and efficient updates to language models, ensuring they remain relevant and accurate in a rapidly changing world. The significant improvement in accuracy on benchmarks like GSM8K suggests that this approach could redefine how knowledge editing is performed in AI systems.
Paper Resources
📖 Reader Mode
~2 min readAbstract:While language models remain frozen at their training state, the world evolves continuously. Knowledge editing has emerged as a key alternative to full retraining, but its deployment is bottlenecked by the erosion of core capabilities: mathematical and programmatic reasoning collapse while encyclopedic recall remains intact. We trace this asymmetric degradation to a distributional mismatch. Covariance-based editors preserve only the subspaces spanned by their reference corpus, but fail to capture the operative distribution shaped by post-training such as SFT and DPO. Static external corpora, including Wikipedia and even the original pretraining mixture, cannot recover this shifted manifold. We propose Moir, which estimates the preservation covariance $C$ directly from the model itself by sampling from its own decoding distribution. Seeding generation with a single random vocabulary token bypasses the instruction-following templates that otherwise dominate sampled outputs, exposing the broader subspaces the model has internalized. Moir requires no external data and serves as a drop-in component for any covariance-based editor, a practical advantage given that the pre- and post-training corpora of most modern LLMs are not publicly accessible. Across OLMo-2, Llama-3.1, and Qwen-3 (7-8B), under both MEMIT and AlphaEdit and in batch and sequential regimes, Moir consistently extends preservation in the most vulnerable domains, most strikingly on Qwen3-8B after 20,000 AlphaEdit batch edits, it retains 79.9% GSM8K accuracy compared to 10.9% with the Wikipedia baseline. These results suggest that aligning the preservation distribution with the model's operative distribution is a key factor in non-destructive editing, and that the model itself may be the most accessible source of that distribution for deployed systems.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.20433 [cs.CL] |
| (or arXiv:2607.20433v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.20433 arXiv-issued DOI via DataCite |
Submission history
From: Jea Kwon [view email]
[v1]
Sun, 10 May 2026 16:05:49 UTC (1,031 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.