Meta Label Correction with Generalization Regularizer

1 Introduction

The remarkable success of deep neural networks (DNNs) in various fields heavily depends on high-quality labels [Algan and Ulusoy, 2021; Wei et al., 2023a; Sukhbaatar and Fergus, 2014; Arpit et al., 2017]. However, many real-world datasets contain noisy labels for different reasons [Hospedales et al., 2021]. For example, in medical datasets, even experienced professionals may occasionally produce noisy labels [Karimi et al., 2020; Ju et al., 2022]. Consequently, mitigating the impact of noisy labels is becoming a critical problem for researchers. Due to the fact that noisy labels can degrade model performance, reducing noise has become one of solutions for addressing the problem of learning with noisy labels. Among these, label correction is very popular since it substitutes original noisy labels with high-confidence pseudo labels to achieve significant model performance.

Label correction methods can be broadly categorized into two types based on the requirements for auxiliary information, i.e., non-self-correction methods and self-correction methods. Existing non-self-correction methods often utilize auxiliary information to replace noisy labels with highconfidence pseudo labels. For example, [Ahn et al., 2023] utilize a pre-trained model to learn effective feature representations for the generation of high-quality pseudo-labels. [Wu et al., 2021] utilize soft labels of meta learning to correct noisy labels with clean samples. Self-correction methods integrate the self-iteration process of the model with deep neural networks to correct noisy labels, without using auxiliary information. For example, [Tanaka et al., 2018] design a specialized network architecture to first estimate and then correct the distribution of noisy labels, while [Li et al., 2022] design a noise correction loss function to correct noisy labels and [Li et al., 2023] utilize noisy tolerant methods under the framework of meta learning to correct noisy labels. Due to the flexibility of noise correction rules, self-correction methods are more popular than non-self-correction methods in real-world applications.

∗Corresponding authors (seanzhuxf@gmail.com).

Although previous label correction methods have achieved significant correction performance, there are still some limitations to be tackled. First, both non-self-correction and selfcorrection methods focus solely on minimizing losses during the optimization process, without considering overall generalization ability of the model. Consequently, this may lead to poor generalization performance. For example, the methods (such as Co-teaching [Yu et al., 2019] and MLNT [Li et al., 2023]) perform well on the training set, but their correction performance significantly decreases wit the increase of the noise ratio, resulting in low generalization ability. Second, current self-correction methods frequently employ lossbased methods (i.e., Beta Mixture Model (BMM) [Arazo et al., 2019] and Gaussian Mixture Model (GMM) [Permuter et al., 2006]) to filter noisy labels, and thus requiring expensive computation resources and receiving limited filtering effectiveness for noisy labels.

To address above issues, we propose a new self-correction method, namely Meta Label Correction with Generalization Regularizer (MLCGR), as shown in Figure 1, to achieve accurate filtering of noisy labels and to enhance generalization ability. To do this, the proposed framework includes the filter and the corrector. The filter uses the gradient score (including the norm and the cosine angle between two adjacent gradients of every sample) to partition the whole dataset into three parts, resulting in accurate data partition and thus exploring the second issue of previous methods. The corrector designs a generalization regularizer into previous meta-learner and base-learner. Moreover, we theoretically demonstrate that our method really achieves the generalization ability, thus exploring the first issue of previous methods.

Compared to previous label correction methods, the main contributions of our method can be summarized as follows:

• The proposed method investigates a new method that allows the meta-based model to automatically correct noisy labels without manual specifying the meta-set.
• We theoretically explain the reasons that affect the generalization ability of the model, as well as design a new regularizer to achieve the generalization ability of the model.
• Extensive experiments on two synthetic datasets with different noise ratios and one real-world dataset validate the effectiveness of the proposed method compared to all comparison methods on classification tasks.

2 Method

Denoting D = {X, Y} = {(xi, yi)}n1 as the original dataset with n samples where xi is the i-th training sample and yi ∈ {0, 1}C is the corresponding label with C classes, denoting ˜y and ˆy, respectively, as the predicted label and the corrected label, the goal of label correction is to train a classification network w with the guidance of a label correction network θ.

2.1 Motivation

Meta-based methods for label correction attempt to replace noisy labels with high-confidence pseudo labels. For example, [Shu et al., 2019] first use Meta Weight Net to relieve negative effect caused by noisy labels, and then assign different values to different losses. [Wu et al., 2021] propose to replace noisy labels with high-confidence pseudo labels directly. However, these methods need to manually specify the clean data as the meta-set for the extraction of metaknowledge. Unfortunately, the meta-set is difficult to obtain in real scenarios. Moreover, the scale of meta-set is quite essential for model performance. In addition, the optimization of existing works only consider the minimization of the loss by ignoring the impact of the generalization ability of the model. To address these issues, we propose the MLCGR method to achieve better correction and generalization performance, as shown in Figure 1.

2.2 Noisy Label Filtering

The goal of filtering methods is to distinguish noisy labels from others. Previous DNN methods tend to fit samples with clean labels fast by setting them with a small loss, compared to the samples with noisy labels [Wei et al., 2023b]. For example, [Arazo et al., 2019] use BMM to distinguish clean and noisy labels by estimating predicted confidence of every sample. [Li et al., 2020] try to fit a two-component GMM to filter clean and noisy labels considering the flexibility in distribution prediction. Although these loss-based methods can effectively filter clean and noisy labels, they still have disadvantages. For example, in the early stage of the model training, clean labels can result in significant loss, but the model may lead to wrong filtering results. Besides, classic methods like GMM require extra memory to make the usage conditions stringent. To address these issues for achieving better filtering performance, we investigate the gradient-based measurement to filter noisy labels.

Given a dataset D with noisy labels, we first divide original dataset into multiple batches. Specifically, we use convolutional neural network (CNN) to train all samples and record their gradient scores (G) during the training process, i.e.,

where m is the number of epochs. To distinguish noisy labels, we consider the change in gradient scores as well as the variations in gradient angles, which is formulated by:

where λs is a non-negative parameter to balance two terms and Norm is the normalization operator.

The first term in Eq. (2) calculates the norm of gradient variations of the same sample in adjacent rounds. We average all gradient variations and normalize them to make it match the second term. After that, the second term calculates the angle between two gradients of the same sample in two sequential epochs to prevent the occurrence of gradient variations, where the value of the norm changes slightly and the direction changes significantly. When all scores are sorted by their values, the higher the score is, the more likely its corresponding label is noisy. Furthermore, we use a non-negative parameter T (0 < T < 1) to divide all samples into three subsets. Specifically, scores smaller than T will be considered as clean samples (i.e., Dcl), scores bigger than 1 - T will be considered as noisy samples (i.e., Dny), and scores in between T and 1 - T will be considered as complex samples (i.e., Dcp). As a result, score calculation by Eq. (2) makes our method early capture the variations of the gradient norm in two consecutive epochs and the changes in the cosine similarity of two gradients in two consecutive epochs. Early detection of gradient variations leads to early detection of noisy labels. Therefore, noisy labels can be accurately filtered and the model requires less memory (verified in Section 3.3). In contrast, other noisy filtering methods, i.e., GMM, filter noisy labels by modeling the distribution of the sample loss, making it difficult to detect noisy labels during the early stage of the training process.

After partitioning the inputted data into three subsets, i.e., Dcl, Dcp, and Dny, by scoring gradient variations, the size of every subset is small. Hence, it is crucial to add the sample size. In this case, data augmentation has played an important role in meta-label correction. However, previous works generally treat all samples equally. For example, [Hussain et al., 2017] use flipping to capture features that appear in various orientations. [Puttaruksa and Taeprasartsit, 2018] apply color jittering to help the model learn more invariant features. However, in these methods, data augmentation makes the samples with noisy labels amplify the noise information as well. In order to address this issue, we investigate to apply different methods of data augmentation to different subsets. Specifically, we apply weak augmentation with flipping and random cropping on complex data, as well as strong augmentation (such as AutoAugment [Cubuk et al., 2018] and RandAugment [Cubuk et al., 2020]) on both the clean data and the noisy data. After this, we obtain the final augmented data with the following form:

In Eq. (3), weak augmentation preserves the fundamental characteristics of the data with noisy labels, but does not introduce significant changes. Hence, it mitigates the disruptive influence of noisy labels on model training. In contrast, strong augmentation enhances data diversity, compelling the model to learn robust features and thus improving its performance. Therefore, applying various augmentation strategies on different data is superior to equally treating all data.

2.3 Meta Label Correction with Generalization Regularizer

After data augmentation, each sample is expected to provide additional useful information for the downstream tasks. These augmented subsets (i.e., ¯Dcl, ¯Dcp and ¯Dny) are then sent to the meta-learner and the base-learner for noisy label correction. However, previous works typically minimize the loss function to obtain high-confidence pseudo labels, which are then used to replace original noisy labels. For example, both MLC and MSLC utilize the meta-learner to train a network of label correction with the help of soft labels and label smoothing. However, they ignore the generalization ability of the model, and thus they can not guarantee robustness of the downstream tasks. To solve this issue, we propose to correct noisy labels by taking the generalization ability of the model into account.

To achieve this, we first follow Model-agnostic Meta Learning (MAML) [Finn et al., 2017], including the inner (i.e., the base-learner Ltr with the parameter w) and the outer (i.e., meta-learner Lme with the parameter θ), to use the following cross-entropy loss ℓ:

where yi denotes the ground truth label and f(xi; θℓ) denotes the predict result. After this, the loss function of the metalearning can be formulated as:

where θ and w are the parameters of the meta-learner and the base-learner.

Since Eq. (5) only takes the original label y into account, it ignores the positive effect of predicted label ˜y. To address this issue, we use Eq. (7) to take both labels into account by setting ˆy0 = y, where ˆy is the correct label and λy is the balance parameter.

After that, Eq. (5) can be rewritten as follows:

In order to make better use of the information after data augmentation, we first regard noisy labels as negative information for the meter-learner, and then rewrite Eq. (6) as:

where λny balances the clean set and the noisy set. Eq. (9) introduces noisy label information into the loss function, which drives the predicted labels far away from incorrect ones to reduce the impact of noisy labels on model performance, and result in learning discriminative features.

Although Eq. (9) can handle the problem of noisy labels, it only focuses on the loss of the model and ignores the model generalization. In this paper, we propose a new regularizer to take the generalization ability of the model into account, i.e.,

where H(Θ) is a function to calculate VC dimension (i.e., dVC) that can bound the generalization ability of the mode and φ is the feature extraction network with parameter Θ. In addition, z1(Θ) is a function to calculate R which is the radius of a sphere, z2(Θ) is a function to calculate M which is the distance between nearest sample and the hyperplane. The variables (such u, v and s) are learnable vectors to help project samples on the sphere. The details of Eq. (10) will be explained in Supplementary Materials Section B.

2.4 Objective Function

By letting LVC = H(Θ) in Eq. (10) and making the expression concise, we set e = φ(x;Θ), so our proposed loss function is formulated as follows.

By minimizing the loss of Ltr and Lme , we obtain the minimized LVC . Furthermore, small dVC can tighten the upper bound of the generalization error, and thus achieving the generalization ability of the model. In this paper, we design a new bi-level minimization method to optimize our proposed objection function, i.e.,

Specifically, we employ Stochastic Gradient Descent (SGD) [Amari, 1993] to alternately optimize Eq. (12). The adaptive optimization process is separated into three steps. In

Algorithm 1 The pseudo-code of the proposed MLCGR.

Input: Training data D = {X,Y}, the batch size, the maximal epoch (Max E), and the maximal correction iteration (Max C).

Output: the parameter of the target classifier network w.

• 1: Initialize the parameters of the meta network, i.e., w(0) and θ(0) , the filtering network θf , and the parameters, i.e., λs, T, λy, λny, and γ.
• 2: while I <Max E do
• 3: G=WarmUp(D, θf).
• 4: Dcl,Dcp,Dny = Score(X,Y,G).
• 5: while t <Max C do
• 6: {(xme cl , yme cl ), (xme ny , yme ny )} = Batch(Dcl, Dny).
• 7: {(xtr cp , ytr cp )} = Batch(Dcp).
• 8: ¯xme = Augstrong(xme ), ¯xtr = Augweak(xtr).
• 9: Update θ(t+1) by Eq. (14).
• 10: Update w(t+1) by Eq. (15).
• 11: Update (˜y(t+1) , ˆy(t+1) with parameters w(t+1) and θ(t+1) .
• 12: end while
• 13: end while

the first step, we extract a small batch of meta samples from Xtr as {(xtr i , ytr i )}n1 i=1 , where n1 is the number of training samples, the parameter w can be updated by the following rule:

After updating the base-learner parameter w(t) to ˆw(t) through Eq. (13), in the second step, we update the parameter θ of the mete-learner by:

where J me i = Lme (f(eme i ; ˆw(t) ), yme i )andandn2 is the number of training samples.

The update of w is under the help of unmodified label correction network, so it may lead to inaccurate pseudo labels. To address this issue, in the third step, we re-update w after the update of θ (i.e., θ(t+1) ), so the specific update rule for w(t+1) is:

By the re-update process, the noise ratio in either the metalearner or the base-learner can gradually reduce. Therefore, noisy labels can be corrected more accurately (verified in Section 3.4). Different from traditional meta label correction methods (such as MSLC and MLC) that solely focus on the smoothness of predict pseudo-labels and the prediction of soft classification outcomes, our proposed method utilizes the information in noisy labels as well as takes the model generalization into account. Obviously, our method achieves effectiveness as well as generalization ability. We list the pseudocode of the proposed MLCGR in Algorithm 1.

3 Experiments

In this section, we conduct experiments on two synthetic datasets and one real-world dataset with different noisy ratios, compared our proposed MLCGR with seven comparison methods, with respect to image classification task in terms of accuracy (ACC).

3.1 Experimental Setup 1

Datasets: We conduct experiments on two synthetic datasets (i.e., CIFAR10 and CIFAR100 [Krizhevsky et al., 2009]) and one real-world dataset (i.e., Clothing1M [Xiao et al., 2015]). Their specific information is summarized in Supplementary Meterials C.1. In our experiments, we follow [Zheng et al., 2021] to corrupt original labels (i.e., CIFAR10 and CIFAR100) with symmetric and asymmetric noise.

Comparison Methods: Seven comparison methods include three meta-based methods (i.e., , MLC [Zheng et al., 2021], MSLC [Wu et al., 2021], and MLNT [Li et al., 2023]), three traditional label correction methods (i.e., Bootstrapping [Reed et al., 2014], M-Correction [Arazo et al., 2019], and Co-teaching [Han et al., 2018]), and the baseline method, i.e., Cross-Entropy [De Boer et al., 2005].

Implementation Datails: We conduct all experiments on a computer with an Intel CPU Core(TM) i9-12900K @3.2 GHz 16-core and a NVIDIA GeForce RTX3090 to implement all methods including the proposed MLCGR with PyTorch framework. We obtain the source code of all comparison methods from the authors, and set the parameters by following the original literature, so that all comparison methods achieve their best performance in our experiments. In the proposed MLCGR, we apply ReLU as the activation function and SGD as the optimizer with a momentum of 0.9. We also let the learning rate as 10-2 and then gradually decay as 10-5 . For noisy filtering, we set the parameters λs and T as 0.3 and 0.1, respectively. For meta label correction, we set the label smooth parameter λy as 0.2, the noisy label parameter λny as 0.1 and the generalization regularizer parameter γ as 0.1. The proposed MLCGR employs ResNet32 as the backbone to extract sample representation for the symmetric noisy and the ResNet-28-10 [Zagoruyko, 2016] for asymmetric noisy. For the real-world dataset Clothing1M, we follow the setting of previous work [Tanaka et al., 2018], and use ResNet-50 a pre-trained model on ImageNet as the backbone. Moreover, we set the number of the warm-up epoch for CIFAR10, CIFAR100 and Clothing1M as 10, 20, and 50, respectively.

1 Related Work and more details of Experimental Setup are reported in Supplementary Materials.

3.2 Result Analysis

We list the classification results of all methods in Tables 1 and 2. Obviously, the proposed MLCGR outperforms all comparison methods on all datasets, followed by Cross-Entropy, Bootstrapping, M-Correction, Co-teaching, MSLC, MLC and MLNT. For example, our method improves an average 3.41% and 13.86%, respectively, compared to the best comparison method (i.e., M-correction) and the worst comparison method (i.e., Cross-Entropy). This indicates that it is necessary to simultaneously filter noisy labels with gradient variations and correct them with the help of generalization regularizer for label correction methods.

First, compared to traditional label correction methods (i.e., Cross-Entropy, Bootstrapping, M-Correction and Coteaching), the proposed MLCGR outperforms them by a large margin. For example, the proposed MLCGR achieves an average improvement of 3.14% over the best correction method (i.e., M-Correction) on all datasets. This verifies the superiority of meta-based label correction methods, as they have good adaptability to various ratios of noisy labels.

Second, compared to meta-based label correction methods (i.e., MSLC, MLC and MLNT), the proposed MLCGR achieves an average improvement of 5.97% over the best correction method (i.e., MLNT) on all datasets. This indicates that the meta-based methods for label correction improve the model performance by achieving the generalization ability of the model.

3.3 Filtering Effectiveness

We investigate the memory consumption and filtering accuracy of GMM on all datasets to evaluate the filtering effectiveness of the proposed MLCGR, and list the results in Figure 2. Obviously, our proposed MLCGR requires less memory and achieves higher filtering accuracy, compared to GMM.

Based on Figure 2a, MLCGR requires an average of 3149.88 KB less memory than GMM. In addition, with increasing epochs, MLCGR is relatively stable, while GMM increases significantly in terms of memory consumption. MLCGR's additional memory consumption scales with sample size, while GMM's is tied to the dimensionality of the data's covariance matrix.

Based on Figure 2b, MLCGR achieves an average increase of 13.62% in noisy label filtering accuracy. Moreover, with the increasing epochs, MLCGR is very stable, while GMM is with fluctuations, in terms of filtering noisy labels. The reason is that the MLCGR method achieves good performance for label filtering performance within a few epochs. In this case, GMM depending on the loss function cannot achieve good performance for label filtering because the loss of clean samples is high during the early states of model training.

3.4 Correction Effectiveness

We report the correction accuracy of all methods on two datasets (i.e., CIFAR10 and CIFAR100) under 20% noisy ratio in Figure 3 2 Obviously, the proposed MLCGR outperforms all comparison methods on these two datasets.

2 Figure 3 does not report the results of Dataset Clothing1M since it has no ground truth.

For example, our method averagely improves by 5.8% and 53.86%, respectively, compared to the best comparison method (i.e., Co-teaching) and the worst comparison method (i.e., MLC). This verifies that our method achieves the best correction accuracy due to its generalization ability. In particular, our method does not achieve good classification performance in the first epochs, such as the first 15 epochs for Dataset CIFAR10 and the first 75 epochs for Dataset CIFAR100. After that, our method achieves good performance. The reason is that 1) in the first epochs our method has a limited number of clean samples to output bad correction accuracy and 2) our method with the increasing clean samples improves the generalization ability of the model with the increasing epochs.

3.5 Ablation Study

The proposed method includes two key components, i.e., noisy label filtering (NLF) in the filter module and the generalization regularizer (GR) in the corrector module. We demonstrate the effectiveness of each component by reporting the classification results of three methods (i.e., NLF without GR in our method, GR without NLF in our method and our proposed MLCGR) in Table 3.

First, the proposed method considering two modules improves on average by 17.71%, compared to the method without considering any module. Moreover, the method without considering any module is worse than the methods considering only one module, i.e., NLF and GR. This verifies that NLF or GR is useful for label correction. In particularly, our method outperforms NLF or GR. This demonstrates that we should consider both NLF and GR for label correction. Second, in two modules, NLF outperforms GR. For example, NLF improves on average about 7.71%, compared to GR, on all datasets. The reason is that accurate filtering and corresponding adaptive augmentation can result in the meta-set with the large sample size. Based on Lemma B.1 in the Supplementary Materials, the increasing samples can reduce the upper bound of the generalization error. Hence, NLF is able to improve the generalization ability of the model.

3.6 Parameter Analysis

In Eq. (2), we employ parameters λs and T, respectively, to achieve a trade-off between gradient norm and direction, as well as to divide all samples into three subsets. In experiments, we investigate their variations by setting λs ∈ [10-3 , 103 ], T ∈ [0.1, 0.7], λny = 0.1 and γ = 0.1 and report the results in the upper row of Figure 4. Obviously, our MLCGR obtains good performance if λs is smaller than 0.1 and T is smaller than 0.3. However, if the value of λs is greater than 1 and T is greater than 0.5, our MLCGR achieves bad performance. This demonstrates that large values of T and λs, respectively, result in more noise labels and recusing the filtering accuracy, and thus affecting the label correction.

In Eq. (11), we employ parameters λny and γ, respectively, to enrich the training information and control the generalization ability. In experiments, we investigate their variations by setting λny ∈ [10-3 , 103 ], γ ∈ [10-3 , 103 , λs = 0.1 and T = 0.1. We report the results in the bottom row of Figure 4. Obviously, the proposed MLCGR delivers superior performance when λny and γ are smaller than 10-1 . However, if the values of λny and γ are greater than 100 , the proposed MLCGR cannot achieve satisfactory performance. This indicates that the effectiveness of the Cross-Entropy loss may be effected with the large values of λny and γ, thus failing to provide sufficient high confidence labels for the model.

4 Conclusion

In this paper, we proposed an effective meta-based label correction method to address the issues in previous methods.

Specifically, we investigated a filter to first partition the entire dataset to three subgroups based on the gradient scores and then to adaptively conduct data augmentation on every subgroups. for achieving better filtering subsets. We also deigned a new generalization regularizer into both the metalearner and the base-learner to achieve the generalization capability of the model. Experimental results on real data sets verify the effectiveness of the proposed method in terms of different classification tasks in different noise ratios.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFA1004100, the National Natural Science Foundation of China under Grant 62306066.