"i Forgot About You": Exploring Multi-Label Unlearning (MLU) for Responsible Facial Recognition Systems

Abstract

The widespread adoption of machine learning and deep learning models has heightened privacy concerns, as these models can unintentionally memorize and expose personal information. Machine Unlearning (MU) has gained considerable attention for improving privacy and data control. MU addresses privacy challenges by selectively removing the influence of specific training data from deployed models. However, most current MU approaches focus on single-label classification scenarios, where each instance is assigned only one label. In contrast, Multi-Label Classification (MLC), such as those in facial recognition (facial attribute classification) systems, involve instances that can be associated with multiple, non-exclusive attribute labels. The complex interdependencies between parameters in these cases pose unique challenges when selectively removing specific knowledge. This work proposes a novel parameter space-based MU framework for MLC systems. Our data-driven generalization approach uses sparsification techniques operating directly on learned representations without retraining on the modified training data. We employ two strategies to improve state-of-the-art models for MLC unlearning: Weight Filtering, which identifies and resets critical parameters based on sensitivity and influence scores, and Weight Pruning, which strategically eliminates parameters based on their importance to the unlearned label while preserving shared representations for retained attributes. Extensive experiments demonstrate that our Weight Pruning method can achieve up to 35.5 $\times $ × speedup over retraining while maintaining >93% accuracy for retained labels and reducing the prediction of forgotten attributes to near zero (0.11%), a significant improvement over existing methods. The privacy analysis also confirms a substantial reduction in information leakage, which establishes a new standard for responsible facial attribute classification systems under current privacy regulations.