COLUR: Confidence-Oriented Learning, Unlearning and Relearning with Noisy-Label Data for Model Restoration and Refinement

1 Introduction

AI models have achieved remarkable success in computer vision tasks due to the availability of massive datasets and advanced computational resources. In general, the performance of a trained model is heavily dependent on the quality of labeling on training data. The models need to keep learning new knowledge from the data generated every day. However, massive label noise inevitably occurs with the new data. As a result, model updating on the new data with heavy label noise will inevitably lead to performance degradation.

According to the analysis above, label noise is a major factor contributing to the degradation of model performance. As shown in Figure 1, the label noise in a dataset mainly comes from two sources: (a) mislabeled data from human or automatic annotators, namely explicit label noise; (b) hard labels on ambiguous data, namely implicit label noise. Obviously, it is not practical to manually correct and refine the labels from a huge amount of training data. Moreover, fully retraining a degraded task model from scratch is also time-consuming and computationally expensive. To mitigate the influence of label noise, especially explicit label noise, Learning with Noisy Labels (LNL) [Algan and Ulusoy, 2021], is a highly relevant research area that has attracted more attention in recent years. The goal of LNL is mainly to train the model from scratch on the basis of data with noisy labels. As stated above, since retraining a model from scratch is generally impossible in realworld situations, we aim to restore and refine a trained model that has suffered degradation.

∗Corresponding author

In general, LNL methods can be grouped into two categories. One is the model-based approach, in which various LNL model architectures are constructed for label correction [Li et al., 2021; Ortego et al., 2021; Zhang et al., 2021b], and sample selection [Li et al., 2020; Cheng et al., 2021; Karim et al., 2022]. However, this approach is not suitable for real AI systems because different task models may have different architectures and have been fine-tuned for specific tasks. As a result, the essential requirement is to refine the performance of the degraded task models instead of replacing them with LNL models. The second category is the modelfree approach to improve the robustness of the model against label noise by robust losses [Liu et al., 2020; Xu et al., 2019; Wang et al., 2019] or regularizers [Englesson and Azizpour, 2021; Sarfraz et al., 2021]. However, this approach is typically designed for the model learning process over noisy label data, which is invalidly used for the restoration of the degraded task models. Although existing LNL approaches have limitations in model restoration and refinement (MRR), the underlying building blocks can be easily borrowed to construct the proposed MRR framework.

To efficiently implement MRR, we need to mitigate the influence on a model w.r.t. those noisy label data and relearn the data with refined labels. Inspired by neuroscience research [Gong et al., 2024; Bao et al., 2025], Forgetting has been experimentally demonstrated for efficient relearning after learning [Ryan and Frankland, 2022]. We argue that Machine Unlearning (MU) [Xu et al., 2023] is a machine forgetting [Sha et al., 2024] mechanism that provides a possible way to unlearn the influence of label noise on degraded models, avoiding retraining the model from scratch. After unlearning, it is easier and more effective to relearn the model using LNL techniques. In fact, the "Learning, Unlearning and Relearning (LUR)" theory has demonstrated its success in various areas, including education [Klein, 2008], self-training [Dunlap and Lowenthal, 2011], and online learning for deep neural networks [Ramkumar et al., 2023; Miao et al., 2024]. As a result, we propose a Confidence-Oriented LUR (COLUR) framework to efficiently and effectively address the challenges of MRR over noisy label data.

Specifically, the proposed COLUR is a model-agnostic MRR framework, which aims to refine the confidence over noisy label data samples for training AI models with different architectures. Inspired by the co-training architecture [Han et al., 2018; Yu et al., 2019; Jiang et al., 2018] in LNL, we additionally duplicate the task model architecture and initiate a teacher model. Given any degraded model learned on noisy label data, we first perform machine unlearning to reduce the model confidence on those data with high disagreement scores measured by teacher and student models. Then we relearn the unlearned model over an augmented dataset by mixing low-confidence label data with high-confidence label data [Carratino et al., 2022; Zhang et al., 2018]. Through the unlearning and relearning process, COLUR can effectively restore and refine model performance. The contributions of this work are summarized as follows:

• We insightfully study a critical but inadequately investigated problem of model restoration and refinement (MRR) on widespread noisy label datasets.
• We propose a robust MRR framework, COLUR, in view of the "learning, unlearning and relearning" mechanism that benefits relearning frommisinformation.
• We implement COLUR with a co-training architecture to iteratively refine model confidences to unlearn the impact from noisy labels and select confident labels for relearning.
• Extensive experiments are conducted on a collection of real datasets to demonstrate the superior performance of COLUR in restoring the degraded models and to visualize the results of label correction.

2 RelatedWork

LNL and MU methods are two key references for COLUR.

2.1 Learning with Noisy Labels

LNL mainly aims to train a model based on data with noisy labels [Algan and Ulusoy, 2021], which can be divided into two types of approaches. One is the model-free approach to improve the robustness of the model against label noise by robust losses [Ghosh et al., 2017; Xu et al., 2019; Zhang and Sabuncu, 2018; Zhang et al., 2021a], such as GJS [Englesson and Azizpour, 2021] using the generalized Jenson-Shannon divergence as a loss, or regularization methods [Zhou et al., 2021; Sarfraz et al., 2021], such as ELR [Liu et al., 2020] adopting the early-learning regularization strategy. The other is the model-based approach for label correction and sample selection. Co-training is one of the representative methods to conduct label refinement in terms of agreements and/or disagreements between twomodels. In thisway, Co-teaching [Han et al., 2018] and Co-teaching+[Yu et al., 2019] train two DNNs, where Co-teaching selects training samples with agreement, and Co-teaching+ selects samples with disagreement based on the idea of Decoupling [Malach and Shalev-Shwartz, 2017]. DivideMix [Li et al., 2020] uses Gaussian mix models to partition the original dataset, employing the semi-supervised technique MixMatch to generate mixed training data. JoCoR [Wei et al., 2020] extends Coteaching with a data augment strategy to align teacher and student models with agreements. Label correction focuses on correcting the labels of noisy samples. NLNL [Kim et al., 2019] employs negative learning to refine the classification boundary of noisy label data. DISC [Li et al., 2023] proposes a dynamic threshold strategy that focuses on the memory strength of DNN to select and correct for noisy labels.

In view of the inefficiency of LNL for MRR, we propose COLUR to address the challenges of MRR in terms of the "learning, unlearning and relearning" mechanism.

2.2 Machine Unlearning

Machine Unlearning (MU) refers to the process of selectively removing the influence of specific training data from a trained model [Warnecke et al., 2021; Graves et al., 2021; Golatkar et al., 2020; Becker and Liebig, 2022; Izzo et al., 2021]. Model fine-tuning (FT) [Warnecke et al., 2021] achieves unlearning by fine-tuning the retained dataset, but may lead to catastrophic forgetting [Kirkpatrick et al., 2017]. Gradient ascent (GA) [Graves et al., 2021; Golatkar et al., 2020] reverses the model training process by moving the model parameters in the direction where the loss of erased data increases. Typical methods can remove the influence of the specified sample on the model parameters by modifying the fisher information matrix [Becker and Liebig, 2022] or the influence function (IU) [Koh and Liang, 2017; Izzo et al., 2021]. In addition, the ℓ1-sparse based weight pruning (L1-WP) adopted to improve the sparsity of the model could improve the effectiveness of data erasure [Liu et al., 2024]. Some recent work has explored more precise unlearning on target forget instances. Unlearning via Null Space Calibration (UNSC) [Chen et al., 2024] constrains the unlearning process within a null space tailored to the remaining samples to ensure that unlearning does not negatively impact the performance of the model. SalUn [Fan et al., 2024] introduces the concept of "weight saliency" to narrow the performance gap with exact unlearning.

Although MU methods can be used as a tool to mitigate the influence of label noise for MRR, they are limited in practice because they require prior clarification of which data points need to be deleted. In addition, unlearning processes can unintentionally degrade model performance and can lead to "over-unlearning" [Chen et al., 2024].

3 ProposedMethod

3.1 Preliminaries

Problem Formulation

Consider a training dataset D0 = {(x0i , y0 i )}Ni=1 , where x0i ∈ X 0 are input samples and y0 i ∈ Y 0 are the corresponding labels. A task model f(θ0 ) is trained by minimizing the following classification loss:

Now, let us denote a new training dataset with noisy labels Du = {(xui , yu i )}Mi=1 where each yu i ∈ Y u may be an incorrect label on xi. Then, a degraded model f(θu ) can be obtained by incrementally trained on Du :

The goal of this paper is to find an optimal θu rl that can restore and refine the performance of the degraded model f(θu ) based on Du :

Definitions

Model confidence [Kim et al., 2019; Li et al., 2023] and label smoothing [Szegedy et al., 2016] play an important role in later sections, so we define them first.

Definition 1. (Model Confidence): Given a model f(θ), and a sample x, the confidence of the model c(θ) w.r.t. x is defined as:

p is the probabilities over all classes output by f(θ).

Definition 2. (Label Smoothing): Given a hard label y, the label smoothing function is defined as:

where y denotes the one-hot vector of y, 1 stands for an allone vector, K is the number of label classes, and γ ∈ [0, 1] is the smooth rate.

3.2 Framework Overview

Figure 2 shows the workflow of COLUR. First, given a task model f(θ0 ) that was originally trained on a dataset D0 (cf. Eq (1)), the incremental "learning" process is conducted on a real-world dataset with label noise, and leading to f(θu ). If significant performance degradation (e.g. 10%) occurs in f(θu ) (cf. Eq (2)), we start the "unlearning and relearning" process to restore the performance of the degraded model, and finally obtain the refined task model f(θu rl ) (cf. Eq (3)). In particular, we adopt a co-training strategy to conduct the unlearning and relearning process, where f(θt ) serves as the teacher model and f(θu ) serves as the student model.

3.3 Details of COLUR for MRR

In this section, we present the details of the "Learning, Unlearning and Relearning (LUR)" scheme for MRR.

(1) Learning on Noisy Label Data

We can obtain the performance-degraded model f(θu ) if the task model f(θ0 ) is incrementally trained on the dataset Du (cf. Eq (2)), i.e. learning on noisy label data as shown in Figure 2 (b).

(2) Unlearning Impact of Noisy Label Data

According to the LUR theory [Klein, 2008; Dunlap and Lowenthal, 2011; Ramkumar et al., 2023], unlearning aims to eliminate highly uncertain knowledge and assumptions to allow new insights, while relearning builds on previous experiences. Therefore, we need to identify the most likely mislabeled data and perform unlearning to mitigate the influence on the degraded model f (θu ) and then relearning on the labelrefined data.

Data with High-Confidence Disagreed Labels. To identify the data associated with high-confidence label noise, we first collect predictions (labels and probabilities) from both the teacher model f(θt ) (note that θt is initialized with the copy of θ0 ) and the student model f(θu ) with input X u :

Then, we extract the disagreement sets according to predictive labels from the teacher model and the student model:

If the confidence scores c(x; θt ) and c(x; θu ) (cf. Eq (4)) of both models are higher, the disagreement on x tends to have a higher confidence. Accordingly, we define the joint confidence score in terms of their geometric mean:

Intuitively, the data samples with higher confidence disagreement indicate higher uncertainty, i.e., higher probability of noise. As a result, we can obtain the high-confidence noisy label datasets, S u τ , S t τ , at level τ:

Unlearning High-Confidence Noisy Label Data. According to the LUR theory, we need to mitigate the influence of the above high-confidence noisy label data, S u τ , from the degraded model f(θu ). Label smoothing (LS) is an effective tool for dealing with overconfidence and label noise [Szegedy et al., 2016; Lukasik et al., 2020; Wei et al., 2022]. Recent work [Di et al., 2024] has shown that LS can improve MU by making wrong predictions with equally low confidence. As a result, we adopt an LS-based gradient ascent (GA) [Di et al., 2024] to unlearn the influence of S u τ .

(3) Relearning with Label Confidence Refinement

Relearning after unlearning is an efficient way to rebuild knowledge systems [Ryan and Frankland, 2022; Sha et al., 2024]. Consequently, it is necessary to relearn the new knowledge from the noisy label dataset Du with label refinement based on the above unlearned model f(θu ul ) (cf. Eq (12)).

Data with High-Confidence Agreed Labels. The agreement datasets can be obtained as Eq (7):

The joint agreement model confidence score on input x is computed by:

Similarly to Eqs (9) and (10), we obtain the high confidence agreement datasets Atτ and Auτ where c(x; θu ul , θt ) > τ. Furthermore, we define Amix τ : {(x, pmix τ )} by the average probability of Atτ and Auτ , i.e., pmix τ = (pt +pu )/2.

Relearning Data with Low-Confidence Labels. Let us denote the union of the low-confidence disagreement dataset and the low-confidence agreement dataset w.r.t. teacher model and student model as: SAt<τ = S t<τ ∪ At<τ and SAu<τ = S u<τ ∪ Au<τ . For each sample x, we mix the low confidence predictive probabilities of SAt<τ and SAu<τ , and obtain SAmix <τ : {(x, pmix <τ )}:

where βm ∼ Beta(αm, αm). Recent work has shown that Mixup [Zhang et al., 2018] is an effective regularization technique to deal with label noise [Carratino et al., 2022] defined as follows, where β ∼ Beta(α, α).

To refine the data with low-confidence labels, we mix them with the data with high-confidence labels.

Then, we relearn teacher and student models with the above mixup data based on the unlearned model f(θu ul ):

Relearning Data with High-Confidence Agreed Labels. In general, the data with high-confidence agreed labels, i.e. Atτ and Auτ , imply correct labels with high probability. As a result, they can serve as high-quality representative samples for relearning. To avoid over-confidence in these data, we relearn Amix τ : {Xτ , Yτ} with label smoothing:

As shown in Figure 2, the steps (2) Unlearning and (3) Relearning are run alternately. More details about the algorithm of COLUR can be found in the online extended version.

4 Experiment

4.1 Experiment Setups

Dataset Preparation. We use four datasets in this experiment: CIFAR-10 (C-10) [Krizhevsky et al., 2009], CIFAR100 (C-100) [Krizhevsky et al., 2009], Flower-102 (F102) [Nilsback and Zisserman, 2008], and Oxford-IIIT Pet (P-37) [Parkhi et al., 2012]. CIFAR-10 and CIFAR-100 consist of 60,000 low-resolution images classified into 10 and 100 classes, respectively. Flower-102 has 8,189 highresolution images in 102 classes, and Oxford-IIIT Pet contains 7,349 high-resolution images of cats and dogs in 37 classes. These datasets cover both low- and high-resolution data, providing a comprehensive evaluation.

As shown in Table 2, we use the official split of the train set Dtr and test set Dts in the dataset package. Each train set Dtr is further divided into initial training data D0 and incremental training data Du . Du is further divided into a noisy label subset Du n and a clean label subset Du c , with noise ratios. For noise types, symmetric noise is applied to CIFAR-10 and Flower-102 by randomly changing labels to any other class. In CIFAR-100 and Oxford-IIIT Pet, we apply asymmetric noise by changing labels to another within the same superclass, simulating real-world mislabeling. This setup ensures both diversity and balance in noise types in datasets. More details can be found in the online extended version.

Comparison Methods. To comprehensively compare the performance of MRR, a set of representative SOTA LNL methods are involved, including (1) Robust Loss Approach: GJS [Englesson and Azizpour, 2021]; (2) Regularization Approach: ELR [Liu et al., 2020]; (3) Co-training Approach: Decoupling (Decoup.) [Malach and ShalevShwartz, 2017], Co-teaching (CoTe.) [Han et al., 2018], Coteaching+(CoTe.+) [Yu et al., 2019], and JoCoR [Wei et al., 2020]; (4) Label Correction Approach: PENCIL [Yi and Wu, 2019], NLNL [Kim et al., 2019] and DISC [Li et al., 2023];.

Training Details. Since the proposed COLUR framework is model-agnostic, we use EfficientNet [Tan and Le, 2019] and WideResNet [Zagoruyko and Komodakis, 2016] as the backbone for different datasets, as detailed in Table 2. To better train different comparison models, we try the AdamW and SGD optimizers with a momentum of 0.9 and a weight decay of 0.001. Other more detailed settings can be found in the online extended version. For each comparison model, we carefully tuned its hyperparameters.

4.2 Comparison with SOTAMethods

Quantitative Results

To comprehensively compare the robustness between the proposed COLUR and different types of SOTA LNL methods as presented in the last subsection, we respectively set the noise ratios of labels at 25% and 75% (normal case), and 10% and 90% (extreme case). In addition, we also reported the performance of Degrade models after training on Du as the reference to illustrate the capability of model restoration. According to the results reported in Table 1, we observe that the performance of all comparison methods generally decreases as the noise ratio increases. As expected, a higher level of label noise leads to higher uncertainty, increasing the difficulty of model classification. In particular, our COLUR method consistently achieves better performance across all noise levels compared to other comparison methods.

For example, COLUR achieves the best MRR accuracy of 87.74% (CIFAR-10) and 92.26% (Oxford-IIIT Pet), outperforming other baselines such as the second-best methods, DISC 85.15% (CIFAR-100) and NLNL 90.43% (Oxford-IIIT Pet) at a noise ratio of 75%. On other datasets with different levels, COLUR also demonstrates the superiority of our method in handling various levels of label noise. We can find that all the comparison LNL methods tend to perform better at lower noise ratios but struggle as the noise increases, especially in the case of asymmetric noise, where label corruption is more systematic. Overall, COLUR has experimentally demonstrated higher robustness and effectiveness across different noise levels, further confirming its applicability in real-world scenarios with highly noisy label data.

Visualization of Label Correction

Figure 3 visualizes the accuracy after MRR for comparison methods, where each axis represents a different noise ratio (25% and 75%) with respect to each dataset. From the radar charts, it is evident that the proposed COLUR method consistently maintains a larger area compared to other baseline methods, indicating the superiority of label noise correction for all datasets and noise levels. As presented above, the comparison LNL methods show decent performance at lower noise levels, but their effectiveness diminishes as the noise ratio increases. These visualizations further confirm that the proposed COLUR is more robust and effective across various datasets at different levels of label noise, highlighting its advantage in label correction in terms of the LUR mechanism.

Further Analysis on Extreme Cases

To assess the robustness of our methods and other baselines, we further analyze all comparison methods under extreme label noise conditions: one is the case of very low label noise with a ratio of 10% and the other is the case of very high label noise with a ratio of 90%. The results of four datasets have been reported in Table 1.

Specifically, in the case of a very low label noise ratio (10%), most comparison methods perform reasonably well, as the majority of the data are clean. However, our method still achieves the highest accuracy, which indicates that COLUR can not only correct explicit mislabels, but also refine implicit label noise on ambiguous images. In the very high label noise ratio (90%), the performance of most LNL methods deteriorates significantly due to the overwhelming presence of noisy labels. In particular, COLUR still maintains better performance compared to other methods, demonstrating its robustness in handling extremely noisy label datasets. This suggests that our unlearning and relearning strategy effectively mitigates the impact of severe label noise.

Visualization of Confusion Matrices. The confusion matrices in Figure 4 show the classification performance in noisy label sets Du n for different methods at different noise ratios (10% and 90%). At a very low noise level (10%), most methods maintain a high true positive rate, as seen by the strong diagonal in their matrices, although some show minor misclassifications. However, COLUR stands out by maintaining a cleaner diagonal with fewer off-diagonal entries, indicating its superiority under minor label noise.

At a high level of noise (90%), most methods experience a significant increase in off-diagonal elements, signaling higher misclassification rates, while COLUR still maintains a relatively strong diagonal with fewer misclassifications than other methods, highlighting its superior ability to correct labels and recover model performance under extreme noise conditions. This performance gap demonstrates the effectiveness of COLUR in mitigating the impact of label noise, surpassing other methods, and achieving a higher true positive rate under extremely high noise levels.

4.3 Comparison between LNL andMU for MRR

According to the LUR theory, unlearning can eliminate the impact of false information. Here, we illustrate the efficacy of MU methods, including GA [Graves et al., 2021], IU [Izzo et al., 2021], L1-WP [Liu et al., 2024] and SalUn [Fan et al., 2024], by comparing them with LNL methods. In particular, the ground-truth noisy label datasets Du n are used as the forgetting set for unlearning.

Quantitative Results

Table 3 shows the performance of all the comparison methods with 50% label noise. By comparing Original with Degrade, we can find that performance degradation is significant (∆Acc is above 20%) after incremental training on Du . The degradation is more significant on datasets with asymmetric label noise, such as CIFAR-100 and Oxford-IIIT Pet.

LNL methods generally outperform MU methods, although MU methods use the ground-truth noisy label data Du n for unlearning. This is because MU can only mitigate the influence of these noisy label data while no additional corrected knowledge can be incorporated into the degraded models, whereas LNL methods can effectively learn with noisy-label data to restore degraded models with corrected pseudo labels. Especially, DISC achieves the overall best performance in all LNL methods thanks to the data augmentation mechanism that improves the robustness to noise.

COLUR takes advantage of both the LNL and the MU methods, so it consistently outperforms the baseline methods across all datasets, achieving results that surpass the Original models in most cases. This demonstrates the efficacy of the inspiration mechanism, namely "relearning after unlearning". The unlearning step effectively mitigates the influence of noisy label data recognized by the co-training models. COLUR performs relearning on the refined labels with improved model confidence.

Error Labeling Rate After MRR

To evaluate the label correction capabilities of each comparison method, Figure 5 demonstrates the error labeling rate on the noisy label subset Du n after performing model restoration on CIFAR-10, CIFAR-100, Flower-102, and Oxford-IIIT Pet. Lower error rates indicate higher capability for label correction and more robustness to noisy labels. From Figure 5, we find that COLUR achieves overall lower error rates compared to the LNL and MU methods in all datasets and noise types. In particular, MU methods underperform other types of methods because MU methods only reverse the influence of noisy label data, but they are incapable of correcting those labels. Therefore, MU cannot be used for MRR independently.

4.4 Ablation Study

We conduct ablation studies on the three key modules of the COLUR framework, the unlearning module (UL, cf. Eq (12)), the Mixup module (MP, cf. Eq (17)), and the label smoothing module for relearning (LS, cf. Eqs (20) and (21)) to evaluate their individual contributions and the effectiveness of their combined usage. Experiments are conducted on the Flower-102 dataset with 50% symmetric noise and the Oxford-IIIT Pet dataset with 50% asymmetric noise. The results presented in Table 4 demonstrate the impact of each module and highlight the performance gains achieved through their integration. By comparing the usage of a specific module, we can find that the performance is correspondingly different. The best results are consistently achieved by the full model, which proves that the design of the COLUR scheme is the most effective for MRR.

5 Conclusion

This paper addressed the issue of model degradation due to learning over widespread noisy label data. Inspired by the theory of "learning, unlearning and relearning (LUR)", we propose the COLUR framework to address this promising challenge. Following the principle, the proposed COLUR framework iteratively refines the confidences of the model through machine unlearning and relearning, which can effectively restore and refine the degraded task models on noisy label data. Extensive experiments are conducted on four real datasets of various aspects. All results consistently show the superiority of COLUR compared to other SOTA comparison methods in restoring the degraded model and label correction.

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (Granted No. 62276190).

Contribution Statement

Zhihao Sui and Liang Hu contributed equally to this work.