Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
Quick Answer
This paper shows that Self-Generated Text Recognition (SGTR) finetuning effectively prevents and reverses emergent misalignment (EM) in models like GPT-4.1 and Qwen2.5-32B-Instruct, outperforming benign baselines.
Quick Take
This approach enhances character alignment without worsening any metrics, indicating that character fortification is key to prevention.
Key Points
- SGTR finetuning shows consistent reduction in emergent misalignment across multiple datasets.
- All interventions achieved comparable EM reversal, but SGTR uniquely prevented misalignment.
- Removing identity-bearing prompts significantly lessens the effects of EM finetuning.
- Emergent misalignment destabilizes aligned character rather than adopting a coherent misaligned persona.
- Three models were tested: GPT-4.1, Qwen2.5-32B-Instruct, and Seed-OSS-36B-Instruct.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits, suggesting that EM operates through disruption of the model's aligned character rather than direct learning of harmful content. Motivated by this connection, we study self-generated text recognition (SGTR) finetuning as a character-targeted intervention that is distinct from existing in-training defenses. We conduct two-stage finetuning experiments across three models (GPT-4.1, Qwen2.5-32B-Instruct, Seed-OSS-36B-Instruct) and multiple EM datasets to compare SGTR finetuning against benign finetuning baselines (correct domain-specific data, general knowledge, and word counting) to find it an effective defense in both reversal and prevention settings. We find that all interventions produce comparable EM reversal, but only when restoring capabilities that EM had degraded. For prevention, only SGTR finetuning consistently reduces misalignment without exacerbating any individual metric, suggesting that character fortification specifically drives prevention. We provide further evidence for EM's relation to the LLM's default character by showing that EM finetuning induces diversity into the LLM's identity self-reports, artificially corrupting self-recognition exacerbates misalignment caused by EM finetuning, and that removing the model's identity-bearing system prompt substantially reduces the effect of EM finetuning. Together, these findings reframe EM not as the adoption of a coherent misaligned persona but as the destabilization of aligned character.
| Comments: | 18 pages, 11 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.23700 [cs.CL] |
| (or arXiv:2606.23700v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.23700 arXiv-issued DOI via DataCite |
Submission history
From: Arush Tagade [view email]
[v1]
Thu, 4 Jun 2026 00:04:58 UTC (2,132 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.