DeepClean: Machine Unlearning on the Cheap by Resetting Privacy Sensitive Weights Using the Fisher Diagonal

Abstract

Machine learning models trained on sensitive or private data can inadvertently memorize and leak that information. Machine unlearning seeks to retroactively remove such details from model weights to protect privacy. We contribute a novel, lightweight unlearning algorithm that is directly applicable to a diverse range of model architectures, from traditional Convolutional Neural Networks (CNNs) to cutting-edge Vision Transformers. This direct applicability distinguishes our approach from previous methods that necessitate exhaustive retraining or large matrix inversions, thereby offering a more efficient solution. Our algorithm judiciously utilizes the diagonal elements of the Fisher Information Matrix (FIM) to selectively update weights, facilitating the effective erasure of sensitive data subsets while minimizing the impact on the retained data. However, the crux of our innovation is not the FIM itself, but the algorithm’s capacity to approximate the complete FIM while significantly reducing computational overhead. Experiments show that our algorithm can successfully forget any randomly selected subsets of training data across neural network architectures. By leveraging the FIM diagonal, our approach provides an interpretable, lightweight, and efficient solution for machine unlearning with practical privacy benefits.