SELC: Self-Ensemble Label Correction Improves Learning with Noisy Labels

1 Introduction

The recent success of deep neural networks (DNNs) for vision tasks owes much to the availability of large-scale, correctly annotated datasets. However, obtaining such highquality datasets can be extremely expensive, and sometimes even impossible. The common approaches, such as web queries [Li et al., 2017] and crowdsourcing [Song et al., 2019], can easily provide extensive labeled data, but unavoidably introduce noisy labels. Existing studies [Arpit et al., 2017; Zhang et al., 2021] have demonstrated that DNNs can easily overfit noisy labels, which deteriorates the generalization performance. Thus, it is essential to develop noise-robust algorithms for learning with noisy labels.

Given a noisy training set consisting of clean samples and mislabeled samples, a common category of approaches [Reed et al., 2015; Arazo et al., 2019; Zhang et al., 2020] to mitigating the negative influence of noisy labels is to identify and correct the mislabeled samples. However, the correction procedure in these methods only updates the noisy labels using the model prediction from the most recent training epoch directly, thus it may suffer from the false correction as the model predictions for noisy samples tend to fluctuate. Take a bird image mislabeled as an airplane as an example. During the training, the clean bird samples would encourage the model to predict a given bird image as a bird, while the bird images with airplane labels regularly pull the model back to predict the bird as an airplane. Hence, the model prediction gathered in one training epoch may change back and forth between bird and airplane, resulting in false correction.

We investigate the reason for performance degradation by analyzing the memorization behavior of the DNNs models. We observe that there exists a turning point during training. Before the turning point, the model only learns from easy (clean) samples, and thus model prediction is likely to be consistent with clean samples. After the turning point, the model increasingly memorizes hard (mislabeled) samples. Hence model prediction oscillates strongly on clean samples. Triggered by this observation, we seek to make the model retain the early-learning memory for consistent predictions on clean samples even after the turning point.

In this paper, we propose self-ensemble label correction (SELC), which potentially corrects noisy labels during training thus preventing the model from being affected by the noisy labels. SELC leverages the knowledge provided in the model predictions over historical training epochs to form a consensus of prediction (ensemble prediction) before the turning point. We demonstrate that combining ensemble prediction with the original noisy label leads to a better target. Accordingly, the model is gradually refined as the targets become less noisy, resulting in improving performance. However, it is challenging to find the turning point. Existing works estimate the turning point based on a test set or noise information, which are unobservable in practice. We propose a metric to estimate the turning point only using training data, allowing us to select a suitable initial epoch to perform SELC. Overall, our contributions are summarized as follows:

• We propose a simple and effective label correction method SELC based on self-ensembling.
• We design an effective metric based on unsupervised loss modeling to detect the turning point without requiring the test set and noise information.
• SELC achieves superior results and can be integrated with other techniques such as mixup [Zhang et al., 2018] to further enhance the performance.

2 RelatedWork

We briefly discuss the existing methods that do not require a small set of clean data (as opposed to [Xiao et al., 2015]). Robust loss functions. Some methods aim to develop loss functions that are robust to label noise, including GCE [Zhang and Sabuncu, 2018], LDMI [Xu et al., 2019], SCE [Wang et al., 2019] and NCE [Ma et al., 2020]. Loss correction. [Patrini et al., 2017] focus on correcting the loss function explicitly by estimating the noise transition matrix. Sample selection. Co-training-based methods [Han et al., 2018; Lu et al., 2022] maintain two networks, and each network is trained on low-risk samples which are selected by its peer network based on the small-loss criterion. Regularization. These methods [Liu et al., 2020; Lu et al., 2021] prevent memorization of mislabeled samples by using a regularizer. Label filtering. SELF [Nguyen et al., 2020] filters the mislabeled samples by ensemble predictions to improve the performance. Label correction. Joint Opt [Tanaka et al., 2018] and PENCIL [Yi and Wu, 2019] replace the noisy labels with soft (i.e. model probability) or hard (i.e. to one-hot vector) pseudo-labels. Bootstrap [Reed et al., 2015] and Mcorrection [Arazo et al., 2019] correct the labels by using a convex combination of noisy labels and the model prediction. PLC [Zhang et al., 2020] updates the noisy labels of high confident samples with model predictions.

Our method is related to label correction. Compared with existing methods, we focus on using ensemble prediction based on historical model outputs to correct the noisy labels, rather than only using prediction from the most recent training epoch. Our approach is straightforward and yields superior performance. Furthermore, our technique can be employed as an add-on component to further enhance the other approaches in challenging cases.

3 Preliminaries

Supervised Classification. Considering a supervised classification problem with C classes, suppose X ∈ Rd be the input space, Y ∈ {0, 1}C is the ground-truth label space in an one-hot manner. In practice, the joint distribution P over X ×Y is unknown. We have a training set D = {(xi, yi)}Ni=1 which are independently sampled from P. Assume a mapping function class F wherein each f : X → RC maps the input space to C-dimensional score space, we seek f ∗ ∈ F

that minimizes an empirical risk 1N ∑Ni=1 ℓ(yi, f(xi)) for a certain loss function ℓ.

Learning with Noisy Labels. Our goal is to learn from a noisy training distribution Pη where the labels are corrupted, with probability η, from their true distribution P. Given a noisy training set ˆD = {(xi, ˆ yi)}Ni=1 , the observable noisy label ˆ yi has a probability of η to be incorrect. Suppose the mapping function f is a deep neural network classifier parameterized by Θ. f maps an input xi to C-dimensional logits zi = f(xi, Θ). We obtain conditional probability of each class by using a softmax function S(·), thus pi = S(zi). Then the empirical risk on ˆD using cross-entropy loss is

When optimizing Lce by stochastic gradient descent (SGD), the DNNs have been observed to completely fit the training set including mislabeled samples eventually (see Figure 1 (a)), resulting in the test performance degradation in the later stage of training (see Figure 1 (b)).

Noise Models. The generation of real-world label noise is unpredictable, a common methodology to cope with noisy labels is to posit a noise model and design robust algorithms under this model. Then we evaluate the algorithms on the real-world datasets to see their effectiveness. A common noise model is class-conditional noise [Natarajan et al., 2013], wherein label noise is independent of input features and true label is corrupted by either a symmetric or asymmetric noise transition matrix (details are in Section 5.1). Recently, another label noise model, named instance-dependent noise [Zhang et al., 2020; Chen et al., 2021a], is proposed, in which the noise not only depends on the class but also the input feature.

4 Our Method

4.1 Memorization Behavior

Our motivation stems from the memorization behavior of DNNs when trained with noisy labels. In Figure 1 (c), we observe that for clean samples, the model predicts them correctly with the increase of epochs. For mislabeled samples in Figure 1 (d), the model predicts the true labels correctly for most mislabeled samples in the early stage (high blue line), even though the model begins making incorrect predictions because of the memorization of wrong labels (increasing red line). Since the model predictions are relatively correct for both mislabeled and clean samples in the early stage, can these reliable model predictions help correct the noisy labels?

4.2 Ensemble Prediction

To alleviate the impact of noisy labels, existing work Bootstrap [Reed et al., 2015] proposes to generate soft target by interpolating between the original noisy distributions and model predictions by βˆ y+(1-β)p, where β weights the degree of interpolation. Thus the cross-entropy loss using Bootstrap becomes

However, applying a static weight (e.g. β = 0.8) to the prediction limits the correction of a hypothetical noisy label. Although another work M-correction [Arazo et al., 2019] makes β dynamic for different samples, the one-step correction based solely on the model predictions at the most recent training epoch still easily incurs false correction.

Since the predictions gathered in a single training epoch for correction is sub-optimal, we generate the ensemble prediction ˜ p for each sample, aggregating the predictions over multiple previous epochs by exponential moving average. Let's denote the model prediction in epoch k as p[k-1]. In epoch k, we have ensemble prediction

where 0 ≤ α < 1 is the momentum. Based on the Eq. (3), we can derive the ensemble prediction in k-th epoch as ˜ p[k] = ∑kj=1 (1 - α)αk-j p[j]. Although ensemble prediction requires a new hyperparameter α and auxiliary memory to record, it maintains a more stable and accurate prediction, especially for mislabeled samples.

4.3 Self-Ensemble Label Correction

We seek to utilize the ensemble predictions to progressively enhance the targets in loss function. There are two options to be considered.

• Option I. Directly use ensemble prediction as the target.
• Option II. Preserve the original noisy label, and combine it with ensemble prediction as the target.

The first option is widely adopted in semi-supervised learning, as the ensemble prediction learned from the labeled inputs can be used as targets for the unlabeled inputs. However, in the noisy labels setting, the model needs supervisions from noisy labels as no extra clean samples are provided. We would compare these two options in Section 5.4. In SELC, we choose the second option. Specifically, for each training sample, we initialize the soft target t using original noisy label ˆ y. Then we update t in each training epoch k by

Based on the Eq. (4), we rewrite above equation as

The first term preserves the original noisy labels with exponential decaying weights αk . The second term is exactly the ensemble prediction at epoch k. Therefore, in training epoch k, the loss of SELC becomes

where pi[j] denotes the model prediction in epoch j for input xi. The first loss term is actually the cross-entropy loss Lce but weighed by αk . With the increase of training epoch k, αk becomes smaller. Thus Lselc is less and less reliant on original noisy labels. The second loss term maintains an exponential moving average of historical prediction as target, and penalizes model predictions that are inconsistent with this target. As a consequence, SELC effectively prevents memorization of mislabeled samples (low red line in Figure 1 (f)) and attains superior performance.

4.4 Estimation of Turning Point

In semi-supervised learning, the initial epoch to perform ensemble prediction is not crucial as the supervision from the clean set guides the model to predict consistent prediction throughout the training. Comparatively, in our scenario, the model would overfit to noisy labels, causing the model predictions to deteriorate. Therefore, it is essential to select the initial epoch in SELC before the turning point T , at which the model starts to memorize mislabeled samples.

We can clearly observe the occurrence of turning point by monitoring the test accuracy drop (Figure 1 (b)). However, the test set is unobservable in practice. The way to accurately identify the turning point without a test set and noise information remains challenging and underexplored.

In this paper, we propose three metrics and choose the optimal one to estimate the turning point by modeling training samples' loss distribution without requiring a clean test set. Due to the memorization behavior of DNNs, the clean samples tend to have smaller loss values than the mislabeled samples in early stage. We analyze the normalized loss distribution over different training epochs in Figure 2 top row. Intriguingly, the two distributions are merged at the initialization, then start to separate, but resume merging after the turning point. Therefore, we propose to estimate the turning point by finding the epoch that has the largest distance between two distributions. To model these two distributions, we use two unsupervised learning approaches: Gaussian Mixture Model (GMM) [Permuter et al., 2006] and K-Means.

epoch 1

�

�

�

Metric 1 and Metric 2. We fit a two-component GMM to loss distribution (in Figure 2 bottom row). The probability density function (pdf) of GMM with M components on the per sample loss value ℓ can be defined as P(ℓ) = ∑Mm=1 πmN(ℓ | µm, σ2m), ∑Mm=1 πm = 1, where πm is the coefficient for the linear convex combination of each individual pdf N(ℓ | µm, σ2m). We use the ExpectationMaximization (EM) algorithm to estimate the πm, µm and σ2m. For Metric 1, we directly calculate the distance between two components by

For Metric 2, we calculate the Kullback-Leibler (KL) divergence of two components as distance.

Metric 3. We fit two clusters by K-Means on the loss distribution. Then we calculate the distance between two cluster centroids S1 and S2 as the Metric 3.

When we train the DNNs with noisy labels, we monitor these three metrics. Once they achieve the maximum value, the corresponding epoch is likely to be the turning point T. We compare three metrics on CIFAR-10 with label noise in Figure 3. M1 is the most reliable and stable one since its corresponding epoch of maximum value precisely aligns with the epoch when test accuracy starts to drop in Figure 1 (b) in all noise cases. We put pseudocode of SELC in Algorithm 1.

5 Experiments

This section, first, investigates the effectiveness of the proposed SELC for classification with class-conditional noise

Algorithm 1 SELC pseudocode.

Input: DNNs f(Θ), training data ˆD = {(xi, ˆ yi)}Ni=1 , Estimated turning point T , total epoch Tmax, hyperparameter α Output: Optimized DNN f (Θ∗ )

• 1: Let t = ˆ y.
• 2: Select an initial epoch Te < T (e.g. Te = T - 10).
• 3: while epoch e < Tmax do
• 4: if epoch e < Te then
• 5: Train f(Θ) by CE loss in Eq. (1) using SGD.
• 6: else
• 7: Update t by (5).
• 8: Train f(Θ) by SELC loss in Eq. (6) using SGD.
• 9: end if
• 10: end while

(Section 5.1), instance-dependent noise (Section 5.2) and real-world noise (Section 5.3). This is followed by several empirical analyses (Section 5.4) to shed light on SELC.

5.1 Class-conditional Label Noise

Datasets and Networks. We conduct the experiments with class-conditional label noise on CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2009]. Given these two datasets are initially clean, we follow [Patrini et al., 2017] to inject noise by label transition matrix Q, where Qij = Pr[ˆ y = j | y = i] denotes the probability that noisy label ˆ y is flipped from true label y. We evaluate SELC in two types of noise: symmetric and asymmetric. Symmetric noise is generated by replacing the labels for a percentage of the training data with all possible labels uniformly. Asymmetric noise is designed to mimic the structure of real-world label noise, where the annotators are more likely to make mistakes only within very similar classes (e.g. deer → horse and cat ↔ dog). We use the ResNet34 [He et al., 2016] as backbone for both datasets, and train the model using SGD with a momentum of 0.9, a weight decay of 0.001, and a batch size of 128. The network is trained for 200 epochs. We set the initial learning rate as 0.02, and reduce it by a factor of 10 after 40 and 80 epochs. We fix hyperparameter α = 0.9. More discussions on α are in Section 5.4. Note that we do not perform early stopping since we don't assume the presence of clean validation data.

All test accuracy are recorded from the last epoch of training.

Improving Other Methods. In comparison to the original noisy labels, we obtain cleaner targets t after using SELC. Therefore, SELC can be easily integrated with other methods. For fair comparison with the existing approaches (e.g. Mcorrection and NLNL) that use mixup or multiple stages of training to boost the performance, we propose SELC+ which uses the corrected labels from SELC to retrain an initialized DNNs using mixup.

Results. Table 1 shows the results on CIFAR with different types and levels of class-conditional label noise. SELC achieves excellent performance compared to the methods that only modify the training loss without extra techniques to boost the performance. When integrated with mixup data augmentation, SELC+ achieves the best performance across most noise ratios, demonstrating the effectiveness of the proposed method on class-conditional label noise.

5.2 Instance-dependent Label Noise

Datasets and Networks. We follow the recent works SEAL [Chen et al., 2021a] and PLC [Zhang et al., 2020] to inject instance-dependent label noise to CIFAR. SEAL generates the controllable label noise based on the assumption that 'hard' (low confidence) samples are more likely to be mislabeled. PLC introduces Polynomial Margin Diminishing (PMD) noise which allows arbitrary noise strength in a wide buffer near the decision boundary. For fair comparison with SEAL, we use the same network architecture Wide ResNet28×10. As for PMD noise, we use the same network architecture PreAct ResNet34 as PLC.

Results. Table 2 shows the results on instance-dependent label noise from SEAL. Our approach consistently achieves the best generalization performance over different noise ratios. The larger the noise ratio is, the more improvement SELC obtains. Table 3 lists the performance of different methods under three types of PMD noise at noise level 35% and 70%. We observe that the proposed method outperforms baselines across different noise settings. When the noise level is high, performances of a few baselines deteriorate and become worse than the standard (CE) approach. In contrast, the improvement of SELC is substantial (∼10% in accuracy) for the more challenging CIFAR-100 with 70% label noise.

5.3 Real-world Label Noise

Datasets and Networks. We use ANIMAL-10N [Song et al., 2019], Clothing1M [Xiao et al., 2015] and Webvision [Li et al., 2017] to evaluate the performance of SELC under the real-world label noise settings. ANIMAL-10N contains human-labeled online images for 10 animals with confusing appearance. The estimated label noise rate is 8%. Clothing1M consists of 1 million images collected from online shopping websites with labels generated from surrounding texts. The estimated label noise rate is 38.5%. WebVision contains 2.4 million images crawled from the web using the 1,000 concepts in ImageNet ILSVRC12. The estimated label noise rate is 20%. For ANIMAL-10N, we use VGG-19 with batch normalization. For Clothing1M, we use ResNet50 pretrained on ImageNet. For Webvision, we use InceptionResNetV2. Note that all the compared method do not use mixup to boost the performance for fair comparison.

Results. Table 4, Table 5 and Table 6 show the results on ANIMAL-10N, Clothing1M and Webvision respectively. On ANIMAL-10N and Webvision, our approach outperforms the existing baselines. On Clothing1M, SELC achieves the comparable performance to PLC, despite its simplicity.

5.4 Empirical Analysis

Correction Accuracy. The key idea of SELC is to correct the original noisy labels. We analyze the quality of the new target t by calculating its correction accuracy: 1N ∑Ni 1{argmaxyi = argmax ti}, where yi is the true label of xi. Figure 4 (a) shows the correction accuracy of the Option I and Option II (SELC). We observe SELC achieves higher accuracy than Option I and correction accuracy is stable with the increase of training epochs. Figure 5 shows the confusion matrix of corrected labels w.r.t the true labels on CIFAR-10 with 40% symmetric label noise. SELC corrects the noisy labels impressively well for all classes.

Hyperparameter Sensitivity. In SELC, we have one hyperparameter α in Eq. (6) to control the momentum in selfensemble. Figure 4 (b) shows the test accuracy when using different α ∈ {0.85, 0.9, 0.95} on CIFAR-10. We observe that the sensitivity to hyperparameter α is quite mild.

6 Conclusion

We propose a simple and effective method SELC to improve learning with noisy labels. SELC leverages the model predictions of previous epochs to correct the noisy labels, thus preventing the model from overfitting to noisy labels. By evaluating SELC on different types of label noise, we observe its superior performance over existing approaches.