How to Make Reproducible Research in Machine Unlearning with ERASURE

1 Introduction

Rapid adoption of machine learning (ML) across industries has brought significant advances in automation, decision making, and data-driven insights. However, it has also introduced challenges related to data handling in compliance with legal and ethical standards. In particular, the right to erasure (or right to be forgotten) in regulations such as the General Data Protection Regulation (GDPR) requires that individuals' data be selectively removed upon request [Mantelero, 2013]. Upon such requests, the model's owner should build a new version without the removed data. However, retraining these models after every request is impractical due to the significant time, economic and environmental costs involved [Crawford, 2022], especially for large models. To overcome this problem, machine unlearning - the process of efficiently removing the influence of specific data points from a model - is emerging as a cornerstone of ethical AI, promoting compliance, accountability, and privacy in data-driven applications.

Machine unlearning is a growing field that currently lacks a widely accepted reference framework for collecting methods, metrics, and datasets. Some studies in the literature have attempted to address this limitation. For instance, in [Choi and Na, 2023], the authors provide the implementation of a limited number of existing unlearning techniques and metrics. However, the repository has not been designed to be an openly available library: the codebase does not allow easy extensions or usage in scenarios other than the replication of the experimental results of the paper. A further attempt at aggregating resources within machine unlearning is presented in [Nguyen et al., 2022], where references to code repositories of the primary unlearning techniques presented in the literature are collected. However, this collection remains fragmented, highlighting the pressing need for a unified framework that provides a standardized interface for seamlessly adopting diverse unlearning techniques and scenarios.

To bridge this gap, we introduce ERASURE 1 , a standardized evaluation framework for the machine unlearning process [Hayes et al., 2024]. It provides ready-to-use implementations of various unlearning techniques, including retrainingbased and approximate methods, along with metrics to assess efficacy, utility, and efficiency [Hayes et al., 2024; Koudounas et al., 2025]. ERASURE also offers curated datasets for benchmarking, ensuring adaptability across various use cases. By integrating existing solutions, datasets, and evaluation measures within a unified framework, ERASURE enhances reproducibility, accelerates advancements, and promotes alignment with regulatory standards.

To showcase ERASURE's flexibility and robustness, establishing it as a reference for future advancements in machine unlearning, this paper first introduces the Machine Unlearning Process, then it presents an overview of ERASURE. Section 3 presents use case scenarios through a main configuration, enabling reproducible experimental evaluations of various unlearning techniques.

1 code available at https://github.com/

aiim-research/ERASURE

2 ERASURE Unlearning Framework

Unlearning Process - The Unlearning Process (UP) typically adopted at production time comprises steps 1 to 4 and uses the already available a and b as shown in Fig. 1. The UP has access to the Original Model b and its Train Set D a (e.g., an image classification model, trained on images published by users on a social network). The UP is triggered when a User's Unlearning Request 1 is received. The Unlearning Request is identified by the Forget Set Df 2 - the specific set of instances that must be unlearned (e.g., all pictures published by a specific user). The Retain Set, Dr △= D \ Df , x is the subset (grey dotted line on the left of Fig. 1) of the Train Set D that must not be deleted. Then, a specific Unlearner (i.e., Unlearning Method) 3 is applied to the Original Model to create the Unlearned Model 4 , which must not have the influence of the Forget set instances. Note that some Unlearners can use only the Forget Set Df while others also use the retain one Dr as depicted by the grey line in the middle of Fig. 1. To have an extended picture of the Machine Unlearning Process, refer to steps x to z as well. In this case, the primary interest is to precisely quantify the quality of the unlearning scenario that researchers and practitioners face (i.e., a company wants to test different unlearners on their datasets and solutions, or a researcher wants to test their method against the SotA ones). To do so, many measures in the SotA rely on the Gold Model y , which is the Original Model trained from scratch on the Retain Set. In the following, we will detail ERASURE design principles and its core components that help researchers and practitioners to meet their goals.

Design Principles - ERASURE is designed to support the needs of researchers and practitioners who aim to test and compare various unlearning techniques across multiple datasets, tasks, and models in a flexible and reproducible way. To meet full extensibility and flexibility, the framework follows object-oriented programming by grounding on abstract classes and fully embracing the factory pattern and the inversion of control paradigms. The companion of these principles is fully configurable workflows - shown in Sec. 3 - that provide robustness and precise control at the same time. The modular structure promotes easy integration of new datasets, unlearning methods, original models, and evaluation metrics. Moreover, to enhance reproducibility, ERASURE includes the implementation of the SotA unlearners, datasets, models, and measures, which can be used out-of-the-box for custom experiments by defining them in the main configuration file.

2.1 Core Components

ERASURE includes components (see Fig. 2) designed to develop and evaluate all the aspects of the unlearning process. Hereafter, we present the three main modules: data management, unlearning methods, and evaluation with measures.

Data Management - The DatasetManager orchestrates data handling by creating the DataSource, which is responsible for loading data from a specific source (e.g., files or repository). ERASURE provides built-in support for all the datasets available through the widely-used libraries HuggingFace [HuggingFace, 2025], TorchVision [TorchVision, 2025], and UCI Repository [Kelly et al., 2025]. Once loaded, the data undergoes a series of - built-in or custom - preprocessing steps that are applied on the fly as batches are loaded. Developers can seamlessly integrate their own preprocessing logic by porting their existing code into ERASURE. ERASURE's vision is that the data can be partitioned through a cascade of configurable DataSplitters. ERASURE already provides the splitters that solve the typical selection scenarios, e.g., by percentages, based on class groupings, or by a fixed sample count. This versatility enables users to define data partitions (such as training, testing, forget, or retain sets) to suit their specific experimental protocols without coding.

Unlearning Methods - The Unlearner class encompasses the common logic of each unlearning strategy. In ERASURE, we specifically included the most adopted and widely recognized Unlearners from the literature to evaluate the key dynamics of the machine unlearning field, ranging from retraining-based methods to efficient approximate techniques such as selective gradient dampening and synaptic decay: Gold-Model, Fine-Tuning, Successive Random Labels, CFk [Goel et al., 2022], EU-k [Goel et al., 2022], NegGrad [Golatkar et al., 2020], Advanced NegGrad [Choi and Na, 2023], UNSIR [Tarun et al., 2023], Bad Teaching [Chundawat et al., 2023a], SCRUB [Kurmanji et al., 2024], Fisher Forgetting [Golatkar et al., 2020], Selective Synaptic Dampening [Foster et al., 2024], and Saliency Unlearning [Fan et al., 2023]. Moreover, the 13 available Unlearners have been implemented and refactored, with the possibility of chaining (i.e., by adopting the Cascade class) them together or applying different combinations of them, e.g., using any unlearning method by exploiting the saliency maps generated by SalUn.

Evaluation - The Evaluator module incorporates all the necessary components for assessing the different kinds of performance of unlearning techniques, such as efficiency and efficacy. To facilitate large-scale experiments, the Evaluator Manager automates the setup and evaluation process by initializing all the required Measures (see its abstract class) and by executing them sequentially. A shared Evaluation object collects the results of the evaluated measures. The evaluation starting point (see the Runners measure class) might include efficiency performance metrics such as the running time, the FLOPS (floating point operations), and memory usage (implemented through PAPI [ICL, 2024] and torch Profiler, because these metrics must be measured during the running of each Unlearning method. With respect to efficacy, we implemented UMIA (i.e., Unlearning-specific MIA) [Hayes et al., 2024], Relearning Time [Tarun et al., 2023; Golatkar et al., 2021; Golatkar et al., 2020], Anamnesis Index

1 "data": {"class":"<NS>.DatasetManager", 2 "parameters": { "DataSource": {...}, 3 "partitions": ["p_1", ..,"p_n"]}}, 4 "predictor": {"class": "<NS>.TorchModel", 5 "parameters": { 6 "optimizer": {"class": "torch.*.Adam"}, 7 "loss_fn": {"class": "torch.*. CrossEntropyLoss"}, 8 "model":{"class":"<NS>.ResNet18"}}}, 9 "unlearners":[ 10 {"class": "<NS>.GoldModel", "parameters": { ... }}, 11 {"class":"<NS>.AdvancedNegGrad", 12 "parameters": { "optimizer": {...},], 13 "evaluator":{"class": "<NS>.Evaluator", 14 "parameters": { "measures":[ ... ] } }

[Chundawat et al., 2023b] and Adaptive Unlearning Score (AUS) [Cotogni et al., 2023]. For accuracy, all metrics in scikit-learn [Pedregosa et al., 2011] are available.

3 Proof of Concept

The ERASURE users (i.e., researchers and practitioners) can define their experiment through a main configuration file (in JSON notation), which enables the design and execution of complex testing scenarios in a straightforward and reproducible manner. Listing 1 outlines the structure of the main configuration, which consists of four fundamental sections: data, predictor, unlearners, and evaluator. Hereafter, we discuss a specific case study that shows how ERASURE enables an experiment to be made easily and reproducibly.

To do so, we choose to model one of the most referenced but challenging scenarios of the unlearning literature, i.e., we need to unlearn a set of individuals that were part of the training set of a binary classifier able to determine whether the person in the input image is smiling or not. As shown in the main configuration (Listing 1), we defined the ResNet18 model in the predictor JSON object (Line 4) and the CelebA dataset [Liu et al., 2015] as the dataset and we indicated the binary attribute "is smiling" as ground truth. Technically, the dataset and the partitions we want to use are defined in the data JSON object (Line 1 of Listing 1) by specifying the DataSource and the chain of partitions/subsets obtained through the Splitters (see Line 1 of Listing 1 and the details reported in Listing 2). In this case, the CelebA dataset is directly gathered from TorchVision (Line 2) and then divided into four partitions by the following sequence of three splitters: i) (Line 5) the Forget Set is defined through persons' IDs from the whole dataset; ii) (Line 8) the Retain Set (80%) and the Testing Set (20%) are then obtained from the remaining data (other ids) of the previous splitter; iii) (Line 11) finally, the Train Set is obtained by concatenating the Retain and the

1 "DataSource": { 2 "class":"<NS>.TVDataSourceCelebA", 3 "parameters": {"path":"torchvision. datasets.CelebA"}, 4 "partitions": [ 5 {"class": "<NS>.DataSplitterByZ", 6 "parameters": "parts_names":["forget", "other_ids"], 7 "z_labels":[2820, 3227, ... ] }, 8 {"class":"<NS>.DataSplitterPercentage", 9 "parameters": {"parts_names":["ret", " test"], "percentage": 0.8, 10 "ref_data":"other_ids"}}, 11 {"class": "<NS>.DataSplitterConcat", 12 "parameters":{"parts_names":["t", "-"], 13 "concat_splits":["ret", "forget"]}}] }

Forget sets. This sequence of operations is designed to show the potential of ERASURE and guarantee that the Forget Set is not part of the Test Set. The predictor JSON object allows to specify all the model details. In this case, we configured the TorchModel to directly use the PyTorch networks and set all its parameters, like the optimizer and loss function (notably, in Lines 6 and 7, we use the original implementations and parameters without requiring further modifications). The unlearners JSON object (line 9) defines the list of unlearners, which will be evaluated in sequence but in an isolated manner. For example, lines 11-12 of Listing 1 show how to define the AdvancedNegGrad unlearner.. The evaluator JSON object (13) defines the list of measures that must be used with their parameters - omitted here for the sake of space. Once the experiment is defined, it is possible to run it to produce the results that we reported in Table 1.

As proof of concept, all baseline Unlearners maintain the same accuracy as the Original Model but differ significantly in Execution Time. NegGrad is the fastest among them, while re-training the model from scratch (Gold Model) is, as expected, among the slowest.

Acknowledgements

The numerical simulations have been realized on the HPC cluster of the Department of Information Engineering, Computer Science and Mathematics (DISIM) at the University of L'Aquila. This study was carried out within the FAIR - Future Artificial Intelligence Research and received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) - MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 - D.D. 1555 11/10/2022, PE00000013), and it is partially funded by the National Recovery and Resilience Plan (Piano Nazionale di Ripresa e Resilienza, PNRR) - Project: "SoBigData.it - Strengthening the Italian RI for Social Mining and Big Data Analytics" - Prot. IR0000013 - Avviso n. 3264 del 28/12/2021. This manuscript reflects only the authors' views and opinions, neither the European Union nor the European Commission can be considered responsible for them.

Contribution Statement

Andrea D'Angelo and Claudio Savelli contributed equally to this work. Giovanni Stilo is the corresponding author.