Fine-Grained and Efficient Self-Unlearning with Layered Iteration

1 Introduction

The rapid growth of machine learning (ML) applications has revolutionized operational efficiency and user experiences and introduced critical challenges in data privacy and regulatory compliance. The General Data Protection Regulation (GDPR) [Voigt and Von dem Bussche, 2017] emphasizes the Right to be Forgotten, i.e., organizations must remove personal data upon request both precisely and efficiently. This requirement places considerable challenges on data handlers in ensuring effective compliance with privacy protection.

Machine unlearning has emerged as a vital attempt to meet these regulatory demands while mitigating privacy threats such as data extraction attacks [Carlini et al., 2021] and membership inference attacks [Shokri et al., 2017; Hu et al., 2022]. It is still a challenge to efficiently remove the contribution of specific training data from an ML model while maintaining satisfactory model utility. Traditional approaches typically retrain models from scratch using the retaining data only [Nguyen et al., 2022], which achieves perfect forgetting, but such methods are often computationally expensive and impractical in most real-world scenarios [Foster et al., 2024]. To mitigate the unacceptable computational cost, a variety of unlearning methods without retraining have also been proposed to approximate the forgetting effect of model retraining [Kong and Alfeld, 2022; Golatkar et al., 2019; Graves et al., 2020; Chundawat et al., 2022b; Tarun et al., 2021; Chundawat et al., 2022a; Tarun et al., 2022; Foster et al., 2024; Baumhauer et al., 2022], but they often require full access to training data and still incur high computational overheads [Kurmanji et al., 2023].

∗Corresponding Author. Contact xuyun.zhang@mq.edu.au or hongsheng.hu@newcastle.edu.au

To address the limitations stated above, recently proposed unlearning methods [Chen et al., 2023; Cha et al., 2024] finetune the target model with re-labeled forgetting data. The forgetting data are re-labeled to their neighboring classes, and this re-labeling technique can fulfill efficient unlearning promisingly by accessing the forgetting data only. However, existing re-labeling based methods still suffer from the following limitations. While not explicitly specified in their design, they primarily follow a single-step re-labeling scheme where all forgetting data are re-labeled to their neighboring classes only once based on the original model. This scheme fails to consider the individual relationship between a forgetting data sample and the model decision boundaries. As a result, this coarse-grained re-labeling scheme can assign a forgetting sample to a class that is far away from the one it should belong to if retraining is adopted for unlearning. Besides, these methods only re-label the forgetting data with hard labels. This forces the corresponding forgetting data to have a significant correlation only with the re-labeled class but low correlations with the entire class distribution in feature space. Then, only one of the classification regions will significantly affect the decision boundary around the forgetting data. Hence, these methods will result in abrupt reconstruction of the decision boundaries during fine-tuning.

In this paper, we propose an approach named SelfUnlearning with Layered Iteration (SULI) to address the aforementioned limitations and improve unlearning efficacy further. The basic idea is to achieve an iterative fine-grained re-labeling process for the forgetting data, where the model is continuously fine-tuned and employed for label prediction in turn during unlearning. Specifically, the approach consists of two parts: the layered iteration strategy and the selective probability adjustment (SPA). The layered iteration strategy progressively refines the decision boundaries from the high entropy region to the low entropy region. This adaptive approach ensures smooth boundary transitions while preserving model stability and utility. SPA re-labels the forgetting data with the soft-labels by calculating the probability distribution of the corresponding forgetting sample. SULI can substantially maintain model utility, significantly reduce the computational cost, and enhance data privacy preservation.

Contributions.We summarize our contributions as follows:

• We propose a novel unlearning framework named SelfUnlearning with Layered Iteration (SULI). SULI modifies the single-step strategy with the layered iteration strategy, refining decision boundaries by prioritizing highuncertainty samples in the early stages of iteration. The iteration framework achieves smooth and accurate boundary adjustments, improving unlearning effectiveness.
• SULI employs the Selective Probability Adjustment (SPA) strategy to tackle the problem of re-labeling forgetting data with hard labels. SPA calculates a new soft label for each sample to reduce the influence of data instances.
• We evaluate our approaches on three benchmark datasets against the state-of-the-art baselines, using four metrics to assess unlearning effectiveness, utility, privacy, and efficiency, with a detailed ablation study. Experimental results show that SULI achieves superior performance.

2 RelatedWork

Machine Unlearning. Machine unlearning is increasingly essential for ensuring privacy in machine learning models, particularly when data must be deleted due to regulatory requirements and individual privacy concerns [Nguyen et al., 2022]. The existing machine unlearning methods can be divided into two categories:

• Exact Unlearning, introduced by [Cao and Yang, 2015; Bourtoule et al., 2021], ensures the model behaves like the deleted data were never part of training. This is typically achieved by retraining the model from scratch without the unlearned data, producing a parameter state unaffected by the forgetting data. Although effective, this method is computationally expensive and impractical for large-scale models or frequent deletion requests [Kurmanji et al., 2023].
• Approximate Unlearning aims to address these inefficiencies in exact unlearning by adjusting model parameters directly to emulate retraining effects [Izzo et al., 2021; Goldblum et al., 2020; Golatkar et al., 2019]. [Ginart et al., 2019] proposed a probabilistic framework based on differential privacy principles, requiring output distributions of unlearned models to resemble retrained models closely. However, [Thudi et al., 2022] argued that achieving specific parameter configurations does not always guarantee effective

unlearning, as it may fail to remove all traces of the forgetting data. To address this limitation, [Goel et al., 2022] advocated focusing on functional equivalence, which ensures that the unlearned model behaves similarly to the retrained model.

Machine Unlearning in Deep Learning. Unlearning in deep neural networks presents significant challenges due to their high dimensionality and complex architectures [Nguyen et al., 2022]. Existing methods primarily focus on approximate unlearning and vary in reliance on retaining or forgetting data. Table 1 compares applicability to class and instance unlearning scenarios and whether using the forgetting or retaining data. Most existing approaches require access to the full dataset, whereas ours operates solely on forgetting data, supporting both class-wise and instance-wise unlearning.

Most unlearning methods still need access to the original training dataset, using the retaining data to help remove the influence of forgetting data while preserving model utility. Two-stage approaches, such as those by [Kim and Woo, 2022] and [Wu et al., 2022], first unlearn forgetting data and then recover utility using retaining data, achieving robust results but incurring significant computational and storage costs. Methods like Amnesiac Unlearning [Graves et al., 2020] and UGradSL [Di et al., 2024] employ random label assignments or gradient-based techniques to erase forgetting data, while teacher-student frameworks, including SCRUB [Kurmanji et al., 2023] and Bad Teacher [Chundawat et al., 2022a], aim to unlearn data without sacrificing performance. Noise-based approaches, such as error-maximizing noise matrices [Tarun et al., 2021] and Batch Unlearning [Graves et al., 2020], remove targeted gradient updates, and Zeroshot Unlearning [Chundawat et al., 2022b] replaces forgetting data with noise before updating the model. However, reliance on full dataset access imposes a large computational overhead. Other methods only rely on retaining or forgetting data, aiming to reduce computational costs while maintaining model utility. Fisher Unlearning [Golatkar et al., 2019] uses noise updates guided by the Fisher Information Matrix (FIM), which, despite reducing dataset requirements, incurs high computational costs and risks significantly degrading model utility. Similarly, Negative Gradient [Golatkar et al., 2019] applies gradient ascent to erase forgetting data, leading to substantial performance loss.

Recent advancements, including Boundary Unlearning

[Chen et al., 2023] and ADV+IMP [Cha et al., 2024], aim to achieve unlearning by recalibrating decision boundaries based exclusively on forgetting data, thereby eliminating the reliance on retaining data. These methods implement a single-step re-labeling strategy, where forgetting data are reassigned to their nearest class boundary based on their position in the feature space. However, this static approach exhibits significant limitations, as it narrowly focuses on the nearest class and fails to account for the global structure of all classes within the model. Such a constrained perspective often results in the misclassification of forgetting data into irrelevant classes, leading to disruptions in the model's class representation and a reduction in overall performance. This issue becomes particularly pronounced in scenarios with complex decision boundaries or diverse class distributions, where class relationships are intricate and dynamic.

3 Method

3.1 Preliminaries

In our approach, the original training dataset Dtrain = {(xi, yi)}ni=1 consists of samples xi ∈ Rd , representing ddimensional feature vectors, and their corresponding labels yi ∈ {1, 2, . . . ,K}, indicating their class among K unique categories. This dataset encompasses all categories foundational to our model's operation. We designate Df as the subset of data intended for unlearning and Dr as the retaining set for maintaining accurate classification. These subsets are mutually exclusive and collectively exhaustive, satisfying Dr ∪ Df = Dtrain and Dr ∩ Df = ∅. Each sample-label pair from Dtrain is denoted as (x, y) ∈ Dtrain, with (xf , yf ) ∈ Df and (xr, yr) ∈ Dr representing instances within the forgetting and retaining datasets, respectively.

The unlearning objective is to align the output distribution of the unlearned model fwu on the forgetting dataset Df with

Algorithm 1 Self-Unlearning with Layered Iteration (SULI)

• 1: Input: Original model fw0 , forgetting dataset Df , number of layers t, small positive value ϵ
• 2: Output: Unlearned model fwt
• 3: Compute entropy H(xf) for all xf ∈ Df
• 4: Sort Df in decreasing order of H(xf) and partition into t subsets {Df1,Df2, . . . ,Dft}
• 5: Set w0 ← parameters of original model
• 6: for each layer i = 1 to t do
• 7: SPAAdjustment:
• 8: for each x ∈ Dfi do
• 9: Compute P(y|xf ;wi-1)
• 10: Adjust P ′ (y|xf) using SPAwith ϵ
• 11: end for
• 12: Model Update:
• 13: Update wt by minimizing:

• 14: end for
• 15: return fwt

that of the retrained model fwr , specifically aiming for:

Here, fw(x) denotes the model's output probabilities for input x under parameters w. The retrained model fwr is trained exclusively on the retaining dataset Dr, thereby making predictions on Df without prior exposure and relying on generalized knowledge from Dr.

3.2 Challenges of Single-Step Re-labeling

This paper focuses on unlearning in image classification tasks. Current re-labeling based unlearning methods face significant challenges in effectively controlling the impact of forgetting Df . Processing the entire forgetting dataset in a single step often leads to abrupt and inconsistent parameter updates, causing sharp shifts in decision boundaries and compromising model utility. The single-step approaches lack the granularity to refine decision boundaries adaptively, leading to coarse updates and misclassifications, particularly for samples close to the class center.

Figure 1 illustrates this issue. Subfigure (a) shows the model's decision boundaries before unlearning, while (b) depicts the ideal boundaries achieved through retraining on the retaining dataset Dr. In contrast, the single-step approach shown in (c) abruptly adjusts boundaries by re-labeling all forgetting data, resulting in sharp shifts and deviations from the ideal boundaries. Samples near decision boundaries (highlighted in red) are prone to misclassification, weakening the model's ability to generalize and maintain stability.

3.3 Self-Unlearning with Layered Iteration (SULI)

The Self-Unlearning with Layered Iteration (SULI) framework addresses the limitations of single-step re-labeling by introducing a hierarchical iterative process, as illustrated in Figure 2. The unlearning process combines three key components: Layered Iteration Framework, Entropy-Based Prioritization, and Selective Probability Adjustment (SPA). This synergy ensures that the decision boundaries of the model are progressively refined, aligning closely with the ideal retrained state while maintaining stability and generalization. Compared to single-step re-labeling, SULI introduces a structured and layered process to ensure that unlearning progress is in a stable and interpretable manner, avoiding significant disruptions to the overall decision structure.

Layered Iteration Framework. SULI partitions the forgetting dataset Df into L independent subsets based on entropy, ensuring that high-uncertainty samples are addressed first. Each subset Dft is processed iteratively, with the model parameters updated at each step t. The iterative update is guided by the following optimization objective:

where Lt(w) represents the loss function at the t-th iteration. This general form highlights the layered structure of SULI while deferring the specific details of Lt(w) to subsequent sections. As shown in Figure 2, the hierarchical structure ensures that the model's parameters are updated iteratively in a structured manner. The retaining data Dr anchors the decision boundaries, while the forgetting data Df undergoes selective adjustments. This layered approach mitigates abrupt changes and performs a stable transition to boundaries.

Entropy-based Prioritization. To determine the unlearning order of samples in Df , we compute entropy for each Df :

where P(yi | xf ;w0) is the model's predicted probability for class yi. Higher-entropy samples, typically near decision boundaries, are addressed first. Let {Df1, Df2, . . . , Dft} denote the resulting subsets in descending order of entropy. This prioritization directs the model to handle boundary-adjacent samples early, thereby minimizing their influence on subsequent iterations and enhancing the stability and efficiency of unlearning.

Selective Probability Adjustment (SPA). The SPA mechanismmodifies the output probabilities of the model for Df to simulate the effect of unlearning. For each sample xf ∈ Df , the adjusted probability distribution P ′ (y | xf ) is defined as:

where S = ∑y′∈Y\{yf } P (y′ |x; w0) ensures normalization, and ϵ represents the unlearning degree. A detailed discussion on the choice and effect of ϵ is provided in Appendix A. The iterative refinement process is formalized as:

This ensures that the model's parameters wt are updated to minimize the divergence between the adjusted probabilities P ′ (y | xft) and the current model predictions P(y | xft; wt-1). The process is repeated for each subset Dft, progressively refining the model's decision boundaries.

Self-Unlearning (SU). As a baseline, we introduce a straightforward, non-iterative approach, Self-Unlearning (SU), which facilitates the unlearning process by treating the forgetting dataset Df as unseen. This is achieved by modifying each sample's output distribution in a single pass using the SPA algorithm, effectively transforming Df into data that the model perceives similarly to unseen data. In SU, given an instance xf ∈ Df and its adjusted probability distribution P ′ (y | xf ), derived using the SPA algorithm, the SU approach updates the model through a single phrase of fine-tuning. The updated w′ are obtained by minimizing the distillation loss:

where DKL is the KL-divergence, P ′ (y | xf) is the SPAadjusted probability distribution, and P(y | xf ;w0) is the model's output distribution under w0.

4 Experiments

Datasets. We follow the previous works [Chen et al., 2023; Cha et al., 2024] and use three datasets: CIFAR10 [Krizhevsky, 2009], VGGFace2 [Cao et al., 2018], and UTKFace [Zhang et al., 2017]. These datasets are selected to evaluate different unlearning methods, including object recognition, face recognition, and age classification. Detailed dataset information is provided in Appendix B.

Models. Our experiments employ different models suitable for each dataset and train all models from scratch. For CIFAR-10 [Krizhevsky, 2009], we use a ResNet-18 model [He et al., 2016]. For the UTKFace dataset [Zhang et al., 2017], focusing on age prediction and demographic analysis, we adopt the All-CNN model [Springenberg et al., 2014]. For the VGGFace2 dataset [Cao et al., 2018], involving complex face recognition challenges, we utilize a ResNet50 model [He et al., 2016]. Detailed configurations and environment are provided in Appendix C.

Baselines. In this study, we select state-of-the-art machine unlearning methods as our baseline: (a) The Original model is trained on Dtrain. (b) The Retrained model is retrained on Dr from scratch. (c) 'Negative Gradient' (NegGrad)[Golatkar et al., 2019] modifies the original model by fine-tuning on Df through gradient ascent . (d) 'Random Label' (RL) [Graves et al., 2020] assigns arbitrary new labels to Df and fine-tunes the network using these random labels. (e) 'Initial Label' (IL) [Chundawat et al., 2022a] involves re-labeling Df with new labels generated by an initial model and fine-tuning model with new labels. (f ) 'Boundary Shrink' (Boundary) creates adversarial examples from Df and assigns adversarial labels to induce boundary contraction towards disparate classes [Chen et al., 2023]. (g) 'Adversarial Unlearning' (ADV+IMP) [Cha et al., 2024] uses adversarial examples to re-label and weighted importance to update parameters.

Metrics. We adopt the following metrics to evaluate:

• Accuracy on the unlearning and retaining datasets (ADf and ADr ): This assesses the method's capability to unlearn

without compromising the overall model performance.

• Accuracy on the test datasets (ADt ): This metric analyzes the model's generalization performance, further subdivided into unlearning test set (ADtf ) and retaining test set (ADtr ).
• Membership Inference Attack (MIA): MIA assesses unlearning effectiveness by measuring how much information about the forgetting data remains in the model. We perform MIA techniques from [Kurmanji et al., 2023]. Distance metrics like Activation Distance are used to measure unlearning success, but they fail to reflect actual outcomes due to inherent neural network variability [Hayes et al., 2024].
• Runtime: This metric evaluates the method's temporal efficiency, providing a comparative computational cost analysis across various unlearning methods.

Implement Settings and Unlearning Tasks. Our experimental environment includes an NVIDIA RTX 4070 GPU, Python 3.11, and PyTorch 2.1.1. We utilize the ADAM optimizer [Kingma and Ba, 2014] with carefully selected learning rates optimized for both class-wise and instance-wise unlearning tasks. To ensure consistent evaluation, the same set of forgetting data (Df ) is applied across all methods. We perform a grid search (the results are shown in appendix D) to optimize the hyperparameter t within the range [1, 25], selecting t = 2 for all experiments as it balances model utility and unlearning effectiveness. Our experiments cover two primary unlearning scenarios: class-wise unlearning, where early stops when the model's accuracy on Df approaches zero, and instance-wise unlearning, where unlearning ceases when the model's accuracy on Df matches that on a 1% reference dataset. For both scenarios, we designate each class as Df and report averages and standard deviations over five seeds to ensure reproducibility. Detailed implementation setups are provided in Appendix E.

Trade-offs. Achieving effective machine unlearning involves balancing multiple objectives: unlearning quality, model utility, unlearning effectiveness, and runtime efficiency. These objectives inherently involve trade-offs; for instance, maximizing unlearning quality might reduce efficiency or degrade performance on retaining data (Dr) [Kurmanji et al., 2023]. SULI is designed to navigate these trade-offs effectively, ensuring forgetting data (Df ) is treated as unseen while maintaining strong performance on retaining data. It consistently outperforms other methods across multiple metrics (see Figure 3). Detailed discussions are provided in Appendix F.

Class-wise Accuracy. In class-wise unlearning, the goal is to reduce the model's accuracy on forgetting data (ADf ) and ADtf to zero while preserving high accuracy on (ADr ). Table 2 shows that Model Retraining and NegGrad eliminate ADf but harm ADr . Initial Label and Boundary Shrink methods struggle to sufficiently reduce ADf and also degrade ADr . In contrast, ADV+IMP, SU and SULI successfully bring ADf and ADtf to zero. Notably, SULI preserves the highest ADr and maintains the model utility.

Instance-wise Accuracy. In instance-wise unlearning, the goal is to align the model's accuracy on forgetting data (Df ) with its performance on unseen reference data, ensuring fairness and generalizability. For simplicity, small standard deviations are omitted in subsequent instance-wise tables. Table 3 shows that while all methods reduce Df accuracy, SU and SULI outperform others by achieving unlearning targets while maintaining high accuracy on Dr and Dtest.

Privacy Protection and Unlearning Quality Comparison. Effective unlearning is essential for robust privacy, preventing adversaries from using Membership Inference Attacks (MIA) to distinguish removed data from data never in the model. Following Kurmanji et al. [Kurmanji et al., 2023], we train a binary classifier on loss values from Df and Dtest. An accuracy nearing 50% indicates strong privacy protection, as the classifier fails to differentiate between these datasets. To ensure consistent data distributions in class-wise unlearning, Df and Dtf share the same categories. We then use tenfold cross-validation to derive average MIA accuracies, minimizing outlier effects. Table 4 compares unlearning methods across class- and instance-level forgetting, revealing that SULI remains near the ideal 50% attack success rate, thus effectively obscuring data membership. Their hierarchical iteration and Selective Probability Adjustment (SPA) enable controlled, seamless unlearning with minimal utility loss. Further details appear in Appendix G.

Computational Complexity. Table 5 compares the runtime efficiency of various unlearning methods. SULI is the fastest method, requiring significantly less runtime than all other baselines (Table 5). In particular, it is 3 times faster than the adversarial re-labeling methods, which spend substantial time on iterative adversarial example generation. By contrast, SULI updates the model solely based on output distributions, avoiding the costly fine-grained perturbation steps. Overall, SULI provides the best balance of unlearning efficacy, privacy, and efficiency. Let |Df | be the number of samples to be forgotten and K be the number of classes. SULI's total cost can be approximated as O(|Df | ×K), reflecting that it primarily involves computing probabilities and redistributing them for each sample-class pair, rather than performing computationally expensive adversarial example generation. Details are provided in Appendix H.

Visualization of Feature Space. To enable a more intuitive observation of how the decision boundary changes after the machine unlearning, we visualize the decision boundary of the target model's feature space using the tDistributed Stochastic Neighbor Embedding (t-SNE) technique [van der Maaten and Hinton, 2008]. The t-SNE maps high-dimensional data to a low-dimensional space through probability distributions, facilitating the visualization of complex data structures in graphs [van der Maaten and Hinton, 2008]. In figure 4 (d), we observe that the green triangles representing Df are successfully dispersed across adjacent classes, effectively diluting their association with the original class. Concurrently, the clustering of other categories (depicted as dots) remains stable, and the decision boundaries between Dr classes are preserved without significant alteration. Dispersing Df while keeping the decision boundaries of Dr demonstrates the effectiveness of SULI in unlearning target data without compromising the model's utility.

Figure 5: Gradient-weighted class activation mapping of the target models before and after the unlearning.

Class Activation Mapping. Gradient-weighted Class Activation Mapping (Grad-CAM) visualizes the decision-making processes of convolutional neural networks by highlighting image regions that most influence classification decisions [Selvaraju et al., 2020]. Figure 5 compares attention maps across different models. For facial recognition, critical facial features are the eyes and mouth. SULI further disperses attention while concentrating on the face, albeit less on primary features and more on other facial regions. Despite the unlearning processes, the model continues to identify facial features by focusing on less discriminative areas. SULI's facial recognition attention pattern is similar to the retrain model.

5 Conclusion

We have introduced Self-Unlearning with Layered Iteration (SULI), a machine unlearning framework that iteratively refines decision boundaries to remove specified data while maintaining accuracy in retaining data. By circumventing complete dataset access and avoiding abrupt re-labeling, SULI significantly reduces the influence of forgetting data while preserving model utility and privacy. The layered, stepwise design ensures model stability and effectively addresses the limitations of single-step boundary adjustments. In future work, we will explore scaling SULI to larger models and investigate theoretical guarantees for complete unlearning.

Acknowledgments

The work was partially supported by The State Key Laboratory of Novel Software Technology (KFKT2024A03).