Label Distribution for Learning with Noisy Labels

1 Introduction

Deep neural networks (DNNs) are the preferred choices for many classification tasks. A large number of labeled training instances are essential to training DNNs with high performance. It is convenient to obtain enough instances as well as labels with the assistance of the Internet and crawler [Divvala et al., 2014], but noisy labels are inevitable. Training DNNs with noisy labels is challenging since the networks can easily overfit to the corrupted labels [Nettleton et al., 2010].

Many prior works avoid overfitting to the corrupted data by correcting the noisy labels [Yi and Wu, 2019; Hendrycks et al., 2018]. Note that the corrupted dataset inherently contains a large number of samples with clean labels. It is inevitable to make wrong corrections to clean labels due to the uncertainty of the boundary between clean labels and noisy labels. Such wrong operations will result in the decline of the performance. Moreover, current methods are weak at handling various noise patterns. When the noise pattern is changed, some methods will make unstable corrections. As shown in Cross entropy loss Forward correction loss

∗Corresponding author

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

noise ratio

(b) Symmetric noise

Figure 1: Performance comparison for model learned with cross entropy loss on samples with clean labels (i.e., filter out wrong-labeled samples) and model learned with forward correction loss on samples containing noisy labels under different noise patterns on CIFAR10.

Fig. 1, we make a comparison between the model trained with forward correction loss [Patrini et al., 2017], a classical label correction method, on samples containing noisy labels and the model trained with cross entropy loss on filtered samples with clean labels. Forward correction loss has good performances in asymmetric noise pattern but performs poorly under symmetric noise cases. In contrast, the model learned with only clean labels have stable performances on both noise patterns. It shows that the samples with clean labels are more important than the correcting operations under specific noise patterns. However, the uncertainty of labels makes it difficult to identify the samples with clean labels.

To reduce the uncertainty of labels, a metric named label confidence is proposed in this paper for measuring the reliability of each label, in which clean labels get high confidence scores while noisy labels achieve low confidence scores. Note that Label Distribution (LD) naturally provides such a metric [Geng, 2016]. As shown in Fig. 2, LD assigns the description degree dyx to all classes in a distribution format, i.e., dyx ∈ [0, 1] and ∑y dyx = 1. The description degree represents the degree to which y describes x, which is naturally suitable to measure the label confidence. Moreover, compared with directly estimating the confidence score from the feature space, e.g., CleanNet [Lee et al., 2018], the description degree is more reliable because the degree value is restricted by other classes in the distribution format. Motivated by this, this paper proposes a novel algorithm named

1.0

0.8

0.6

0.4

0.2

test accuracy Label Dsitribution based Confidence Estimation (LDCE) to estimate label confidence by generating label distribution.

Note that using some trusted samples is an effective approach to improving robustness in noisy label problems, and such small set can be fetched easily in many real-world applications [Hendrycks et al., 2018]. LDCE estimates the label confidence via a small number of trusted samples, i.e., samples with clean labels. In this case, the training data is divided into two sets, i.e., a set with a few trusted samples and the other set with a large number of untrusted samples. Guided by the trusted samples, LDCE generates LD for each untrusted sample by measuring the similarity in feature space, i.e., the embedding space obtained from a feature encoder. Then, the confidence score of the observed label can be obtained from LD. After obtaining the label confidence, the samples with high confidence scores can be selected from the untrusted set, which is termed as purified data in this paper. Experiments show that the purified data mainly consists of the samples with clean labels.

Since the purified data is selected from the untrusted set, the risk of wrong operations to clean labels is mitigated. In this case, we combine the purified data with the existing correction method to train robust DNNs. The empirical results substantiate that the proposed method achieves favorable performances in both synthetic noise cases and real-world noise cases.

The contributions of this paper are as follows:

• A reliable metric label confidence is designed for measuring the reliability of labels based on label distribution.
• A practical algorithm for estimating the label confidence is proposed. Experimental results verify the efficiency of the estimation algorithm.
• A novel learning method using label confidence is designed for training robust DNNs. Experiments on three datasets show the superiority of the proposed method against state-of-the-art methods.

2 RelatedWork

Due to the presence of noisy labels, most of the learning algorithms based on the supervised learning framework can not accurately capture the mappings between instances and ground-truth labels. To deal with this problem, existing methods focus on mitigating the adverse effect of noisy labels.

One intuitive and easy approach is to remove the samples which are considered as wrong-labeled. For instance, [Han et al., 2018; Chen et al., 2019] attempt to filter out the unreliable samples during the training phase via a co-teaching framework. However, such methods do not explicitly deal with noisy labels. When the noise is severe, the performances of these methods are usually vulnerable.

An alternative approach is to correct noisy labels. [Tanaka et al., 2018; Chen et al., 2019] propose to replace the corrupted label with a more robust soft label which is in a distribution format. By converting a categorical label into a label distribution, the noisy label can be probabilistically corrected. Aside from correcting the labels directly, a loss correction strategy is proposed to revise the effects of noisy labels with the correction loss function [Patrini et al., 2017]. However, wrong corrections to the clean labels will introduce extra noisy information during the learning process.

Other than the works mentioned above, some works notice the value of clean labels and turn to focus on strengthening the importance of the samples with clean labels. In [Guo et al., 2018], CurriculumNet designs a learning schedule which starts from learning 'easy' subset to gradually adding 'complex' subset. In [Ren et al., 2018; Shu et al., 2019], sample reweighting strategy is used to improve the attention of clean labels. Our method belongs to this categroy. However, different from previous methods, we focus on combining the reweighting idea with correction methods.

3 The ProposedMethods

3.1 Notations Definition

First of all, some notations used in this paper are clarified as follows. The i-th instance is denoted by xi. The groundtruth label of the i-th instance is denoted by yi ∈ {0, 1}c and 1Tyi = 1, where c is the number of possible label values and 1 is a vector of all-ones. As the training set is corrupted, the observed label of the i-th instance is denoted by ˜ yi ∈ {0, 1}c and 1T ˜ yi = 1.

In a c-class classification problem, Du = {(xi, ˜ yi)|1 ≤ i ≤ N} is a corrupted dataset, where the observed label ˜ yi is considered as unreliable. Moreover, a trusted dataset is prepared as Dt = {(xi, yi)|1 ≤ i ≤ t}, where t/(t + N) ≪ 1 is defined as the trusted fraction. To generate the label confidence, we introduce label distribution di. The description degree of class j to instance xi is denoted as dji ∈ [0, 1], and ∑cj=1 dji = 1. The label confidence of the i-th sample is defined as ci.

3.2 Confidence Estimation

As refered above, the bottleneck of current methods is the uncertainty on the untrusted set. Measuring the reliability of each label is a practical approach to reduce the uncertainty. Guided by this motivation, this paper designs a metric named label confidence based on label distribution [Geng, 2016] and proposes a practical method LDCE for estimating this metric.

Label Distribution Generation

LD offers a numerical metric description degree for each class in label space. As shown in Fig. 2, a high degree usually denotes more reliable labeling. Thus, the description degree on the observed label is naturally suitable to be the label confidence metric.

Note that samples sharing the similar features tend to have the same label, similarity in feature space has been successfully applied in recovering LD [Xu et al., 2018]. In this paper, the feature similarity is calculated with a small batch of trusted samples. Then, LD is generated according to the similarity scores.

In detail, it is the first step to sample a support set and two query sets from the training data. Sampling subsets to construct meta-task is commonly used in few-shot learning algorithms [Wang and Yao, 2019], which is helpful to learn from limited data. Then, the membership degree [Xu et al., 2018] to class j for instance xi is calculated by

where Sj denotes the samples of class j in the support set, and sij denotes the similarity score between instances xi and xj. Finally, the membership degrees to different classes are normalized into a label distribution di = [d1i , d2i , . . . , dci ] via a softmax layer

After obtaining the label distribution, the label confidence ci is updated iteratively according to

where α is the step size.

Framework of Estimation Model

In order to get accurate label confidence from the corrupted data, it is important to train a reliable feature encoder for similarity calculation. This paper designs a unified learning framework to estimate the label confidence as well as learn a reliable feature encoder. As shown in Fig. 3, the framework is composed of a feature encoder fθ, a metric module g, and a fully connected (FC) layer fφ.

The estimation model is learned with a multi-task strategy [Ruder, 2017], which consists of a metric learning task and a classification task. The output of the feature encoder is denoted as zi = fθ(xi), which refers to the feature embedding of instance xi. Then, the similarity score between two instances is measured as sij = ‖zi - zj‖.

After obtaining the label confidence according to Eq.1-2, the loss function of the metric learning task is calculated by the cross entropy loss on the trusted query set according to Eq. 4.

where m1 denotes the number of the query data sampled from the trusted set.

For the classification task, the linear layer fφ is used to predict the label of instance xi, and the predicted result is denoted as p(yi|xi) = fφ(zi). For data sampled from the trusted set, the loss is calculated directly with the cross entropy loss according to Eq. 5.

where n and m1 denotes the number of support data and query data sampled from the trusted set. For data sampled from the untrusted set, the loss is calculated by Eq. 6 based on the attention mechanism [Vaswani et al., 2017].

where ai denotes the attention value obtained from the label confidence, and m2 is the number of the query data sampled from the untrusted set. Since the encoder is trained from scratch, the estimation result is unstable in the former iterations. In this case, we initialize the label confidence to 0 and calculate the attention value with a threshold δ.

Then, the loss function for the classification task is designed as follows:

where I(.) is the indicator function.

Based on the above analysis, Eq. 4 and Eq. 8 are combined to form the loss function for training the estimation model.

Algorithmic detailes are shown in Algorithm 1.

3.3 Learning with Purified Data

After obtaining the label confidence, the boundary between clean labels and noisy labels becomes clear by assuming that clean labels get higher label confidence. In this case, the samples with high confidence scores are selected with the same threshold in the estimation model, and the selected samples are named as purified data. Then, we combine the purified data with the classical correction method GLC [Hendrycks et al., 2018] by proposing a revised correction loss, and the learning method is named as Purified Data based Loss Correction (PDLC).

Algorithm 1 Label Distribution based Confidence Estimation

Input: Trusted data Dt, Untrusted data Du, fθ, fφ. Parameter: batch size n, m1, m2, max iterations T , step size α, threshold δ.

• 1: Initialize model parameter θ, φ and label confidence.
• 2: for i = 1 to T do
• 3: {x(s) , y(s) } ←SampleMiniBatch(Dt, n)
• 4: {x(q) , y(q) } ←SampleMiniBatch(Dt, m1)
• 5: {x(˜ q) , ˜ y(˜ q) } ← SampleMiniBatch(Du, m2)
• 6: z(s) i ← fθ(x(s) i ), z(q) i ← fθ(x(q) i ), z(˜ q) i ← fθ(x(˜ q) i ).
• 7: Calculate the label distribution di by Eq. 1-2.
• 8: Formulate the learning function by Eq. 4-9.
• 9: Update model parameter θ, φ in backward process.
• 10: Update label confidence by Eq.3.
• 11: end for

Revised Forward Correction Loss

In [Patrini et al., 2017], forward correction loss is proposed to tackle the noisy label problem. The loss function is shown as

where ek denotes the k-th standard canonical vector, i.e., ek ∈ {0, 1}c and 1Tek = 1. C ∈ Rc×c is the noise transition matrix and Cjk = p( ˜ y = ek |y = ej ). How to obtain C is the same as [Hendrycks et al., 2018].

Since the purified data contains mainly samples with clean labels, forward correction loss does not perform well on purified data when compared with cross entropy loss. However, simply replacing the correction loss with the cross entropy loss will mitigate the correction effects on the remaining untrusted samples. In this case, we design a revised forward correction loss by combining forward correction loss with cross entropy loss. The combination loss function is shown as

Specifically, λ ∈ (0, 1) is the hyperparameter to balance the cross entropy loss and the initial correction loss, which is selected according to the model performance on each noise pattern. We observe that the case with a high noise ratio favors a small λ value, while the case with a small noise ratio prefers a high λ value.

Final Objective

After obtaining the purified data, the whole training set is divided into three parts. The first part is the pre-acquired trusted samples Dt with ground-truth labels, and the loss function is the cross entropy loss ℓce. The second part is the purified data Dp with high label confidence, and the loss on Dp is the revised forward correction loss ℓpurified. The last part is the remaining samples with relatively low label confidence that recorded as ˜Du, and the loss on ˜Du is calculated by the forward correction loss ℓcorr.

The DNN model is denoted as fϕ. By minimizing Eq. 15, the optimal parameter ϕ of the DNN model can be obtained.

4 Experiments

4.1 Experimental Setup

Datasets

The experiments are conducted on CIFAR10 and CIFAR100 [Krizhevsky et al., 2009] with synthetic label noise and Clothing1M [Xiao et al., 2015] with real-world label noise.

CIFAR10 & CIFAR100 are two datasets consists of 32 × 32 color images. The two datasets both contain 50,000 training samples and 10,000 test samples. CIFAR10 assigns the samples with 10 classes, while CIFAR100 assigns the samples with 100 classes.

Since a trusted set is essential in the learning settings, the training set is split into two parts with the trusted fraction of 5% and 10%. Then, the synthetic label noise is added into the untrusted set. Following the previous literature [Chen et al., 2019], experiments are conducted on two representative types of label noise: symmetric noise and asymmetric noise. As illustrated in Fig. 4, the label noise is uniformly distributed among all other classes in the symmetric case. In the asymmetric case, label noise is generated by flipping a label to a different class. The noise ratio ε denotes the proportion of wrong labels. In this paper, we test noise ratio 20%, 50% and 80% for both symmetric and asymmetric noise.

Clothing1M is a dataset collected with real-world label noise. The training set consists of 1M images with noisy labels from 14 fashion classes and 47,570 images with manually refined labels. The validation set and test set have 14,313 and 10,526 images respectively. The images with manually refined labels in training set are used as trusted samples.

(a) Symmetric 20%

(b) Asymmetric 50%

(c) Symmetric 80%

Trusted Only Trusted + LDCE

Fine-tuning Fine-tuning + LDCE

Distillation Distillation + LDCE

Symmetric Asymmetric

(d) CIFAR10

Implementation Details

The experiments are implemented with PyTorch framework. Detailed implementations for each dataset are as follows.

CIFAR10 & CIFAR100. For estimation model, we use a ResNet-32 [He et al., 2016] as the feature encoder. The learning rate is 0.1 with a deacy step 60 and a decay rate 0.1. The hyper-parameters is α=0.6, δ=0.5. We observe that the hyper-parameters, which are selected by experience, are not very sensitative to different noise patterns. Thus, the hyper-parameters are fixed for all noise patterns. For classifier model, we adopt a Wide Residual Network [Zagoruyko and Komodakis, 2016] of depth 40 and a widening factor of 2. The learning rate is 0.1 with a multi-step deacy [60, 80, 90] and a deacy rate 0.2. For both estimation model and classifier model, we use SGD optimizer with 0.9 momentum, a ℓ2 weight decay 1 × 10-4 and train the models for 100 epochs.

Clothing1M. Following the previous works [Tanaka et al., 2018; Shu et al., 2019], we use ResNet-50 [He et al., 2016] pre-trained on ImageNet for both the feature encoder and classifier model. The hyper-parameters is the same with CIFAR10 and CIFAR100. The learning rate is 0.01 with a decay step 5 and a deacy rate 0.1. We use SGD optimizer with a momentum 0.9, a ℓ2 weight decay 1×10-3 and train the models for 10 epochs. For preprocessing, we resize each image to 256 × 256, crop the middle 224 × 224 as input, and perform normalization.

Baselines

We compare our algorithm with Trusted Only, referring to learning DNNs with only trusted samples, and Fine-tuning, referring to fine-tuning DNNs trained on corrupted data with trusted samples [Shu et al., 2019]. Other than the two methods above, classical comparison methods with the same learning settings include: Distillation [Li et al., 2017], MentorNet [Jiang et al., 2018], L2RW [Ren et al., 2018], MW-Net [Shu et al., 2019], GLC [Hendrycks et al., 2018].

For fair comparisons, all the contrast methods are evaluated with the same setup. To ensure that the empirical results are reliable, we repeat each experiment on synthetic noise cases 5 times with different random seeds.

4.2 Experimental Results

Results on CIFAR10 & CIFAR100

The label confidence obtained from the estimation model is critical for the learning method PDLC. To investigate the performance of the estimation model, we firstly illustrate the distribution of label confidence in the interval [δ, 1] (δ = 0.5) on CIFAR10 with 5% trusted fraction. As can be seen from the Fig. 5(a)-5(c), for both symmetric and asymmetric noise types, the purified data, i.e., samples with label confidence in the interval [δ, 1], mainly consists of the samples with clean labels. In other words, only limited wrong-labeled samples exist in the purified data, which verifies the effectiveness of the estimation model and the capability of label confidence in identifying the clean labels.

To further investigate the effectiveness of the purified data, we combine the purified data with methods Trusted Only, Fine-tuning and Distillation. Since the three methods are easy to implement, we only need to enrich the trusted samples with purified data. Fig. 5(d)-5(e) summarize the results. It can be observed that the three methods combined with the purified data all achieved performance gain, which verifies the effectiveness of the purified data.

test accuracy

1.0

0.9

0.8

0.7

0.6

0.5

Next, we evaluate the performance of PDLC by comparing the method with seven contrast methods on different noise patterns and trusted fractions. Different from the above experiments that simply enrich the trusted samples with purified data, PDLC leverages the purified data with a revised correction loss function. Table 1 summarizes the experimental results. It can be observed that PDLC achieves favorable performances among different noise patterns.

For symmetric noise cases, PDLC outperforms all the comparison methods in all noise ratios and trusted fractions. When the noise ratio is small (e.g., 20%), most of the comparison methods can achieve high test accuracies. Even in such cases, PDLC still achieves higher test accuracies. When the noise ratio is high (e.g., 80%), the performances of the classifier models drop significantly. In this case, PDLC shows strong superiority over the contrast methods.

For asymmetric noise cases, PDLC also achieves better test accuracies in most cases. Since the forward correction loss used in both GLC and PDLC is well-designed for asymmetric noise, both the two methods can achieve high test accuracies. In this case, the purified data used in PDLC can play a limited role in performance improvements, which explains why PDLC does not rank first in some cases.

Results on Clothing1M

To verify the effectiveness of the proposed method on realworld data, experiments are conducted on Clothing1M, which is a dataset with real-world label noise. We compare PDLC with several methods, including Cross Entropy, Forward [Patrini et al., 2017], LCCN [Yao et al., 2019], PENCIL [Yi and Wu, 2019], MW-Net [Shu et al., 2019] and GLC [Hendrycks et al., 2018]. The results are summarized in Tabel 2. Row 1 to 4 and row 6 are quoted from [Shu et al., 2019], and row 5 is quoted from [Yi and Wu, 2019]. Row 7 to 8 are obtained by our own implementations. It can be observed that PDLC

achieves the best performance against the other methods.

5 Conclusion

In this paper, a novel method LDCE is proposed to estimate label confidence. The label confidence is a metric designed for measuring the reliability of labels. LDCE estimates the label confidence by generating label distribution. Then, the samples with high confidence scores are selected as purified data. To verify the effectiveness of LDCE, we design a learning method PDLC by leveraging the purified data. The experiments conducted on both synthetic and real-world datasets substantiate the superiority of the learning method.

This paper has shown that estimating the label confidence from the corrupted data is a feasible strategy in the noisy label problem. In the future, we will explore more effective approaches for estimating and utilizing the label confidence.

Acknowledgments

This research was supported by the National Key Research & Development Plan of China (No. 2018AAA0100104), the National Science Foundation of China (61622203), the Collaborative Innovation Center of Novel Software Technology and Industrialization, the Collaborative Innovation Center of Wireless Communications Technology, and the National Natural Science Foundation of China (61702095).