RepSelect: Robust LLM Unlearning via Representation Selectivity
Quick Answer
RepSelect introduces a novel approach to LLM unlearning by isolating forget-set-specific representations, achieving a 4-50x greater reduction in post-relearning accuracy compared to five baselines across models like Llama 3 and Qwen 3.5, while maintaining general capabilities.
Key Points
- RepSelect collapses top principal components of weight gradients to enhance unlearning.
- Evaluated on biohazardous knowledge and abusive tendencies across four model families.
- Achieves near-perfect robustness against few-shot prompting attacks.
- Outperforms GradDiff, NPO, SimNPO, RMU, and UNDIAL in unlearning effectiveness.
- Targets selective representations for deeper and more robust forgetting.
Paper Resources
Source Excerpt
arXiv:2606. 17168v1 Announce Type: new Abstract: Making (LLMs) deeply forget specific knowledge and values without sacrificing general capabilities remains a central challenge in unlearning. However, current methods are easily reversed by fine-tuning or few-shot prompting, suggesting their forgetting is only shallow. We identify the root cause.
Existing methods target representations shared with both the retain set and the subspace recovered by a fine-tuning attacker, making unlearning both disruptive to general capabilities and easy to reverse. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.