Rethinking Supervision Targets for Multimodal LLM Unlearning: Less Private Information, Reduced Distribution Shift
Paper · https://arxiv.org/abs/2611.09939
arXiv:2611.09939v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) can memorize and reproduce sensitive information from their training data, posing significant privacy risks. Their ability to encode knowledge across both textual and visual modalities further complicates the selective removal of such private information. Many existing methods guide model unlearning by training the model toward constructed labels or output distributions, which we refer to as supervision targets. However, these methods often suffer from a trade-off between forgetting effectiveness and model utility. We identify supervision target limitations as a key factor in this trade-off and empirically show that targets containing more private information lead to insufficient forgetting, while larger distribution shifts from the original output distribution are associated with greater utility degradation. Motivated by these findings, we propose FFN-Perturbed Self-Distillation (FPD), which starts from the teacher's original output distribution as a zero-shift reference and selectively perturbs FFN activations associated with private information across textual and visual modalities. By precisely reducing private information from the teacher's original output distribution, FPD constructs supervision targets with less private information and reduced distribution shift. Extensive experiments on S-MLLMUn Bench demonstrate that FPD achieves a better forgetting-utility trade-off than existing methods.
First seen by Oriel on .
AI summary
The paper proposes FFN-Perturbed Self-Distillation (FPD), a multimodal LLM unlearning method that reduces private information in supervision targets while limiting distribution shift to improve the forgetting-utility trade-off.
- The authors find that targets containing more private information are associated with insufficient forgetting, while larger shifts from the original output distribution are associated with greater utility degradation.
- FPD starts with the teacher's original output distribution as a zero-shift reference and selectively perturbs FFN activations linked to private information across textual and visual modalities.
- The method uses the perturbed teacher distribution to construct supervision targets with less private information and reduced distribution shift.
- Experiments on S-MLLMUn Bench show a better forgetting-utility trade-off than existing methods.
#multimodal llms #machine unlearning #privacy #self-distillation
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.09939v1 | https://arxiv.org/abs/2611.09939 Read discussion |