Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
Quick Take
The proposed Visual-Noise Guided In-Context Distillation (VGID) framework enhances unlearning in Multimodal Large Language Models (MLLMs) by combining visual perturbation with textual unlearning, achieving a 0.371 reduction in forget set ROUGE-L with only a 0.055 drop in retain set ROUGE-L, addressing privacy concerns effectively.
Key Points
- VGID utilizes dual-modal intervention for effective unlearning in MLLMs.
- Achieves significant unlearning effectiveness while maintaining model utility.
- Reduces forget set ROUGE-L by 0.371 with minimal impact on retain set ROUGE-L.
- Addresses privacy risks associated with sensitive knowledge in MLLMs.
- No external teacher models or explicit annotations are required for VGID.
Article Content
From source RSS / original summaryarXiv:2606. 00105v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable progress on vision-language tasks, but they may also memorize and expose sensitive or restricted knowledge, raising concerns about privacy and broader safety risks. Machine Unlearning (MU) provides a promising way to remove targeted undesirable knowledge from trained models without retraining from scratch while preserving general model utility.
Nevertheless, effective unlearning in MLLMs remains particularly challenging. Existing training-based methods often struggle to balance unlearning effectiveness and model utility. In contrast, training-free methods such as in-context unlearning preserve model utility by avoiding parameter updates, but they do not remove memorized knowledge at the parameter level and may remain vulnerable to reverse-engineering attacks.
More importantly, in-context unlearning is insufficient in multimodal settings, where visual inputs can provide strong conditioning signals and induce undesirable outputs. To address these challenges, we propose Visual-Noise Guided In-Context Distillation (VGID), a distillation-based framework for MLLM unlearning. VGID dynamically constructs an unlearning-oriented teacher distribution from the frozen base model through dual-modal intervention that combines visual perturbation with textual in-context unlearning.
The resulting intervention-induced distribution serves as a teacher signal for distillation, guiding the student model toward parameter-level unlearning without requiring external teacher models or explicit undesirable response annotations. Experimental results show that VGID achieves strong unlearning effectiveness while preserving competitive model utility, reducing forget set ROUGE-L by 0. 371 with only a 0. 055 drop in retain set ROUGE-L in a representative setting.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, enabling efficient fine-tuning with only 0.11% parameter updates. It significantly enhances performance in few-shot learning and domain shifts across 15 biomedical imaging datasets, demonstrating robustness for clinical applications.