ProMix: Combating Label Noise via Maximizing Clean Sample Utility

1 Introduction

The great success of deep neural networks (DNNs) attributes to massive and accurately-labeled training data, which is notoriously expensive and time-consuming to obtain. As an alternative, online queries [Blum et al., 2003] and crowdsourcing [Yan et al., 2014] have been widely used to enhance the labeling efficiency. However, due to the lack of expertise, such annotation processes inevitably introduce wrong labels to models and result in performance degradation. To this end, learning with noisy labels (LNL) [Zhang et al., 2017], which aims to mitigate the side-effect of label noise, has attracted huge attention from the community.

∗Corresponding Author.

Figure 1: (a) Comparison of selected samples for each class at the end of training on CIFAR-10 dataset with 90% symmetric label noise. TP/FP indicates true/false positive. (b) Comparison of pseudo-label recall for unchosen samples for each class at the end of training on CIFAR-10 dataset with 90% symmetric label noise.

A plethora of works has been developed for the LNL problem. Some early approaches design noise-robust loss [Wang et al., 2019; Wei et al., 2022a]. While some other classical methods utilize the latent noise transition matrix to calibrate the loss function [Patrini et al., 2017; Xia et al., 2019; Zhu et al., 2022]. These methods either disregard specific information about label noise or require strong statistical assumptions and tend to underperform under heavy noise and a large number of classes. Although these methods are theoretically grounded, their performance largely lags behind the fully-supervised counterpart.

Another active line of research [Wei et al., 2020; Li et al., 2020a; Yao et al., 2021] attempts to filter out clean samples for training. For example, the pioneering Co-Teaching approach [Han et al., 2018] figures out that clean samples mostly pose smaller loss values due to the memorization effect of DNNs [Zhang et al., 2017]. Later, DivideMix [Li et al., 2020a] further regards the remaining noisy samples as unlabeled and integrates well-developed semi-supervised learning algorithms. To date, this learning paradigm has been the de-facto standard for the LNL task [Yao et al., 2021; Karim et al., 2022; Cordeiro et al., 2023] and shown to excel compared with early approaches. Despite the tremendous success, current selection-based methods solely ask for a decent-enough clean set with a restricted budget to ensure the high-precision of selected samples. By then, an abundance of valuable clean examples gets overlooked, while they possess great potential for performance enhancement.

Motivated by this, we present a novel framework ProMix which comprehensively investigates the opportunities and challenges of maximizing the utility of clean samples in LNL. To achieve this, we first propose a progressive selection mechanism that dynamically expands a base class-wise selection set by collecting those having high prediction scores on the observed labels. Empirically, such a simple modification allows us to reach an extraordinary trade-off between the quality and the quantity of the clean set. As shown in Figure 1a, compared with DivideMix [Li et al., 2020a], our progressive selection strategy selects much more clean samples while rarely introducing detested noise samples. Moreover, a simple baseline that is trained merely on such selected clean samples establishes the state-of-the-art performance on the real-world noisy CIFAR-N [Wei et al., 2022b] benchmark, and incorporating semi-supervised learning can further boost the performance. These results clearly validate the importance of well-utilizing clean samples.

Despite the promising results, this maximal selection procedure inevitably causes side effects for the SSL as well. First, the clean samples may not be uniformly distributed across different labels, leading to a biased label distribution. Second, the selection and joint semi-supervised learning procedures are tightly interdependent, which may suffer from the confirmation bias stemming from its own hard-to-correct errors. To mitigate these problems, we develop a debiased semi-supervised training framework that comprises two key components. The first one is an auxiliary pseudo head that decouples the generation and utilization of unreliable pseudolabels so as to alleviate the model from confirming its error recursively. Second, we incorporate a calibration algorithm that simultaneously rectifies the skewed pseudo-labels and the biased cross-entropy loss. As depicted in Figure 1b, integrated with debiased SSL, ProMix is able to generate pseudo-labels with higher quality on the unselected samples, thus facilitating the model training. By integrating the two key components into a whole framework, our proposed ProMix framework establishes state-of-the-art performance over various LNL benchmarks, even in the imbalanced LNL scenario (see Appendix B 1 ). For example, on the CIFAR-10N datasets with real-world noisy labels, ProMix outperforms the best baseline by 2.04%, 2.11%, and 3.10% under Aggregate, Rand1, and Worst label annotations respectively.

2 RelatedWork

Learning with Noisy Labels. Existing methods for LNL can be roughly categorized into two types. Earlier LNL approaches focus on loss correction. Since the vanilla crossentropy (CE) loss has been proved to easily overfit corrupted data [Zhang et al., 2017], a vast amount of robust losses has been designed to achieve a small and unbiased risk for unobservable clean data, including robust variants of the CE loss [Wang et al., 2019; Xu et al., 2019; Ma et al., 2020] or sample reweighted losses [Shu et al., 2019; Zheng et al., 2021]. Some other methods [Patrini et al., 2017; Xia et al., 2019; Zhu et al., 2022] consider the noise transition matrix and depend on the estimated noise transition matrix to explicitly correct the loss function, but the transition matrix could be hard to estimate in the presence of heavy noise and a large number of classes.

1 https://arxiv.org/abs/2207.10276

The latter strand of LNL methods tries to filter out a clean subset for robust training [Li et al., 2020a; Karim et al., 2022; Cordeiro et al., 2023]. Early methods [Han et al., 2018; Jiang et al., 2018; Wei et al., 2020] typically adopt a peer network and perform small loss selection that is motivated by the memorization effect of DNNs [Zhang et al., 2017]. The seminal work DivideMix [Li et al., 2020a] further incorporates MixMatch [Berthelot et al., 2019] to leverage the unselected examples and obtain extraordinary performance. This paradigm is then followed by the most recent LNL approaches [Li et al., 2020b; Liu et al., 2020; Bai et al., 2021; Karim et al., 2022]. Recent LNL works also incorporate contrastive learning methods [Ortego et al., 2021; Yao et al., 2021]. Despite the promise, existing selection-based methods mostly select a mediocre clean set with medium size and restricted budget, which overlooks a flurry of clean samples and naturally exhibits a quality-quantity trade-off. A too-small budget leads to inadequate clean sample utility, while a toolarge budget enrolls many unexpected noisy examples. To cope with this, our work targets to maximize the utility of clean samples to unleash their great potential.

Debiased Learning. Debiased learning has gained huge attention since the source dataset and algorithms can be naturally biased [Kim et al., 2019; Nam et al., 2020]. For example, in long-tailed learning, models can be easily biased towards dominant classes due to the distribution bias in source data [Zhang et al., 2021a]. As for semi-supervised learning, besides the distribution bias in source data and pseudo-labels [Wang et al., 2022], confirmation bias caused by inappropriate training with unreliable pseudo-labels may jeopardize generalization performance [Chen et al., 2022]. Debiased learning has also been a prevailing topic discussed in some other research topics like contrastive sampling [Chuang et al., 2020] and OOD detection [Ming et al., 2022]. Despite the surge in other areas, the bias in LNL still remains underexplored. In this work, we attempt to alleviate the biases in our selection and pseudo-labeling procedures. We hope our work will inspire future works to integrate this important component in advanced LNL frameworks.

3 ProposedMethod

ProblemFormulation. For a multi-class classification task with C classes, we denote the noisy training dataset with n examples by D = {(xi, ˜ yi)}ni=1 , where each tuple comprises of an image xi ∈ X ⊂ Rd and the corresponding noisy label ˜ yi ∈ Y = {1, ...,C}. In contrast to the supervised learning setup, the ground-truth labels are not observable. The goal of LNL is to obtain a functional mapping fθ,ϕ : X ↦→ Y that takes an example xi as the input and outputs the logits fθ,ϕ(xi) for predicting the true labels. This classification model fθ,ϕ = hϕ ◦ gθ consists of a feature extractor gθ and a classification head hϕ, which are parameterized by θ and ϕ respectively. In our work, we adopt the cross-entropy loss H = - ∑Cj=1 ˜ yi j log pji where pi = softmax(f(xi)) is the

(a) Overall Framework of ProMix.

(b) Progressive Selection.

Figure 2: The left: Overall framework of ProMix. After progressive selection, the source dataset is divided into the labeled set and unlabeled set for debiased SSL. The labeled data are fed forward to both primary head h and auxiliary pseudo hAP, while the unlabeled data are provided with debiased pseudo-labels by h and then propagated forward through hAP. Finally, the standard cross entropy in classification loss is replaced by debiased margin-based loss. FP/BP indicates forward/backward propagation. The right: Details of progressive selection. The MHCS selects those examples with high confidence and matched predictions with given labels to dynamically expand a base selection set.

probabilistic output of f(xi). pji is the j-th entry of pi and ˜ yi j is the j-th entry of one-hot encoded label vector ˜ yi .

Similar to previous works, our framework also divides the samples into a clean set Dl and a noisy set Du, which are then applied for semi-supervised learning. The vanilla classification losses for these two parts are formulated:

where Sharpen(pi, T) = (pci ) 1 T ∑Cc=1 (pci ) 1 T for c = 1, 2, ..,C. An overview of the proposed method is illustrated in Figure 2.

3.1 Progressive Sample Selection

As aforementioned, we would like to excessively collect clean samples. To achieve this, we first draw inspiration from the classical learning theory,

Theorem 1. [Shalev-Shwartz and Ben-David, 2014] Let F be the hypothesis set of f. Denote the population risk by R(f) = E[H] and the corresponding empirical risk by ˆR(f). We define the true hypothesis by f ∗ = argminR(f) and empirical risk minimizer by ˆ f = argmin ˆR(f). Given a dataset containing n examples, with probability at least 1 - δ, the following inequality holds:

where ˆR(F) is the empirical Rademacher of F.

Considering a simple case in which f consists of multiple ReLU-activated fully-connected layers, we can apply the techniques outlined in [Allen-Zhu et al., 2019] to derive an upper bound of the Rademacher complexity O(√ 1n ). If we assume a perfectly clean set with nc samples, the model has larger than a probability of 1 - O(e-ncϵ2 /50 ) to give an ϵwrong prediction on the remaining dirty training data. Hence, this model is prone to produce a much larger probability on the observed labels for the remaining correctly-labeled data than noisy ones. Motivated by this, we present the following selection procedure for enlarging the base selection set.

Class-wise Small-loss Selection (CSS). Based on the small loss criterion, we aim to first filter out a base clean set as the cornerstone. In the training, the patterns of easy classes are usually fitted earlier and the loss values of examples with different observed labels may not be comparable [Gui et al., 2021]. So it would be more appropriate to select examples class by class with the same observed labels. At each epoch, we first split the whole training dataset D to C sets according to the noisy labels, i.e. Sj = {(xi, ˜ yi) ∈ D|˜ yi = j}. Given the j-th set Sj, we calculate the standard CE loss li for each example and select k = min(⌈ nC × R⌉, |Sj |) examples with smallest li to constitute the clean set Cj, where R is the filter rate that is identical for all classes. Finally, the integral clean set is merged as DCSS = ∪Cj=1Cj. Compared to the original small-loss selection, we relate k to the average number of examples n/C such that produce a roughly balanced base set.

Matched High Confidence Selection (MHCS). After getting the base clean set, we also introduce another selection strategy to make capital out of the potentially clean samples missed by CSS. Specifically, we calculate the confidence scores ei = maxj pji for remaining samples. Then, we select those examples whose predictions match the given labels, while their confidence is high enough,

where y′i = argmaxj pji is the predicted label of xi. In practice, we set a high threshold τ such that the selected samples have a high probability of being clean. According to our theoretical motivation, the proposed MHCS mechanism is able to automatically explore new clean data to dynamically enlarge the clean set Dl = DCSS ∪ DMHCS by trusting those examples with matched high confidence.

Remark. Our MHCS selection strategy draws inspiration from the pseudo-labeling procedure of FixMatch [Sohn et al., 2020]. But, in contrast to the SSL setup, our clean set is dynamically generated. With the naive high confidence selection, the DNNs can assign an arbitrary label with high confidence and select them as clean, which results in a vicious cycle. Thus, we choose those samples to have high confidence in their given labels, which tend to be originally clean.

3.2 Debiased Semi-Supervised Training

After sample selection, we would also like to integrate semisupervised learning that utilizes the remaining noisy samples to boost performance. Nevertheless, our excessive selection procedure inevitably induces unexpected characteristics. First, the selected clean samples may inherently be imbalanced amongst different labels, since some labels are typically more ambiguous than others. Hence, the model may be biased towards the classes that possess more clean samples and in turn pollutes the quality of selection (as shown in Figure 1a). Second, the selection and pseudo-labeling are tightly coupled, which naturally brings about confirmation bias, so we should be particularly careful to train the model with the generated pseudo-labels. To escape such a dilemma, we introduce our debiased SSL framework in what follows.

Mitigating Confirmation Bias. In most SSL pipelines, the generation and utilization of pseudo-labels are usually achieved by the same network blocks or between interdependent peer networks. As a result, the mistakes in pseudolabels are nearly impossible to self-correct and can accumulate amplifying the existing confirmation bias. In order to alleviate this, we aim to disentangle the generation and utilization of pseudo-labels, which is achieved via introducing an extra classifier termed Auxiliary Pseudo Head (APH) hAP. The primary classifier h is the task-specific head that generates pseudo-labels for the unlabeled data but is merely optimized with labeled samples in Dl. The newly devised classifier hAP share the same representations with h but have independent parameters. It receives the pseudo-labels from h and is trained with both labeled set Dl and unlabeled set Du. Based on Eq. (1), the classification loss is formulated,

where pi is the softmax score from h and p′i is that from hAP. λu controls the strength of the unsupervised loss. Through this procedure, the backbone benefits from the unlabeled data for enhanced representation via hAP. At the same time, the primary classifier h remains reliable without seeing unlabeled samples, which reduces the confirmation bias. It is worth noting that the auxiliary pseudo head only participates in training and is discarded at inference time.

Mitigating Distribution Bias. In addition to confirmation bias, there may still be distribution bias whose causes are two-fold. First, the selected samples can falsely calibrate the classifier due to skewed label distribution. Second, previous work [Wang et al., 2022] shows that pseudo-labels can naturally bias towards some easy classes even if the training data are curated and balanced, which can iteratively dominate the training process. To learn a well-calibrated classifier, we incorporate a Debiased Margin-based Loss (DML) lDML to encourage a larger margin between the sample-rich classes and the sample-deficient classes. lDML can be defined as:

where α is a tuning parameter that controls debiasing strength and π is the underlying class distribution P(y | x) indicating which label possesses more samples on the training set. Here we assume π is known and will show how to estimate it later. The standard cross entropy loss H is then replaced by lDML.

Similarly, we also resort to Debiased Pseudo Labeling (DPL) following [Menon et al., 2021] to calibrate the pseudolabels. The refined logit ˜ fi which is then fed into the softmax layer for subsequent pseudo-label generation is formulated:

That is, we suppress the logits on those easy classes and ensure other classes are fairly learned. The refined pseudolabels are then used for calculating unlabeled loss in Eq. (4).

Next, we elaborate on our dynamical estimation of π. Notably, we employ two different class distribution vectors for the labeled and unlabeled data respectively. Formally, given a mini-batch of samples B from either Dl or Du, we update the estimation of class distribution π in a moving-average style:

where m ∈ [0, 1) is a momentum coefficient. When calculating the loss in Eq. (4), we use different estimation πl and πu for the two sets. For better understanding of the whole procedure, we refer the readers to Appendix A for more details.

To summarize the pipeline of debiased SSL, the labeled data are fed forward to both h and hAP while the unlabeled data are provided with the debiased pseudo-labels by h as Eq. (6) and then propagated forward through hAP. Finally, the standard cross entropy in Eq. (4) is replaced by lDML. Through the above debiasing procedure, the refined pseudolabels with higher quality and enhanced tolerance for incorrect pseudo-labels assist in milking unlabeled samples and in return enhance the purity of sample selection.

3.3 Practical Implementation

Label Guessing by Agreement (LGA). While we still anticipate selecting matched high confidence instances, there are no free labels to be matched on falsely-labeled data whose predictions mismatch the observed labels. To further distill potential "clean" samples from unlabeled data, we perform label guessing to correct noisy labels on the unlabeled set Du.

We train two peer networks f1 and f2 in parallel to revise the noisy labels by agreement. Then we select a new subset Dc to enlarge the clean set Dl = Dl ∪ Dc by the condition,

In other words, the two networks should both hold high confidence on the same label. After that, we take the shared prediction y′ = argmaxj f j 1 (x) as the guessed labels for Dc. With such reliable refurbishment, more purified clean samples could be involved in Dl to facilitate the SSL training.

Training Objective. To improve the representation ability of ProMix, we involve consistency training along with mixup augmentation on the reliable labeled set Dl. See Appendix A for more details. The losses for consistency regularization and mixup augmentation are denoted as Lcr and Lmix respectively. The overall training loss is given by,

where γ is a weighting factor. Notably, we ensemble the outputs of the two peer networks at the inference time.

4 Experiments

In this section, we present the main results and part of the ablation results to demonstrate the effectiveness of the ProMix framework. We put more experimental results, including results on imbalanced dataset with synthetic noise, results on dataset with instance-dependent label noise and more results for ablation experiment in Appendix B.

4.1 Setup

Datasets. We first evaluate the performance of ProMix on CIFAR-10, CIFAR-100 [Krizhevsky et al., 2009], Clothing1M [Xiao et al., 2015] and ANIMAL-10N [Song et al.,

2019] dataset. For CIFAR-10/100, we conduct experiments with synthetic symmetric and asymmetric label noise following the previous protocol [Tanaka et al., 2018]. Symmetric noise is introduced by randomly flipping each ground-truth label to all possible labels with probability r and asymmetric noise mimics structure of realistic noise taking place in similar classes. We also verify the effectiveness of ProMix on the recently proposed real-world noisy CIFAR-N [Wei et al., 2022b] dataset. The CIFAR-N dataset is composed of CIFAR-10N and CIFAR-100N, which equips the training samples of CIFAR-10 and CIFAR100 with human-annotated real-world noisy labels collected from Amazon Mechanical Turk. We adopt three noise types, Aggregate, Rand1 and Worst with a noise rate of 9.03%, 17.23% and 40.21% respectively for CIFAR-10N and one noise type Noisy-Fine whose noise rate is 40.20% for CIFAR-100N. Clothing1M dataset is a large-scale real-world noisy dataset containing 1 million images crawled from online shopping websites whose labels are collected by extracting tags from the surrounding texts. ANIMAL-10N is also a real-world noisy dataset consisting of human-annotated online images of ten confusable animals.

Implementation details. For CIFAR experiments, we use two ResNet-18 as the backbone of the peer networks. These networks are trained for 600 epochs, with a warm-up period of 10 epochs for CIFAR-10 and 30 epochs for CIFAR-100. We employ SGD as the optimizer with a momentum of 0.9 and weight decay of 5e-4 . The batch size is fixed as 256 and the initial learning rate is 0.05, which decays by a cosine scheduler. The threshold τ is set as 0.99 and 0.95 for CIFAR-10 and CIFAR-100. The debiasing factor α is set as 0.8 and 0.5 for CIFAR-10 and CIFAR-100. m is fixed as 0.9999. We leverage RandAugment to generate a strongly augmented view for consistency training. The filter rate R is set as {0.7, 0.5, 0.2, 0.1} for symmetric noise with noise ratio r = {0.2, 0.5, 0.8, 0.9} respectively and 0.5 for asymmetric noise and all settings On CIFAR-N. LGA is performed at the last K = 250 epochs such that the networks are accurate enough. We also set a maximum ratio threshold of total selected samples ρ = 0.9 for all settings to restrict excessive correction. We linearly ramp up γ from 0 to 1 and λu from 0 to 0.1 to avoid overfitting false labels at the beginning.

For Clothong1M, following previous work [Li et al., 2020a], the backbone is ResNet-50 pre-trained on ImageNet and the training is conducted for 80 epochs. We set the learning rate as 0.002 and weight decay at 0.001. The filter rate R is set as 0.7, K is fixed as 40 and α is set at 0.8. We leverage AutoAugment [Cubuk et al., 2019] with ImageNet-Policy for augmentation. For ANIMAL-10N, we adopt VGG-19 [Simonyan and Zisserman, 2015] with batch-normalization as the backbone following [Song et al., 2019]. We use SGD with the same settings as CIFAR for 300 epochs of training. The filter rate R is set as 0.7. See Appendix for more details.

4.2 Main Results

We compare ProMix with multiple state-of-the-art methods. Specifically, they are Co-Teaching+ [Yu et al., 2019], Mcorrection [Arazo et al., 2019], PENCIL [Yi and Wu, 2019], SELFIE [Song et al., 2019], JoCoR [Wei et al., 2020], DivideMix [Li et al., 2020a], MOIT [Ortego et al., 2021], PLC [Zhang et al., 2021b], NCT [Chen et al., 2021], ELR+ [Liu et al., 2020], JoCoR [Wei et al., 2020], CORES* [Cheng et al., 2021], PES [Bai et al., 2021], SOP+ [Liu et al., 2022], SELC [Lu and He, 2022], ULC [Huang et al., 2022], LongReMix [Cordeiro et al., 2023] and the naive cross-entropy (CE) method. For their performance, we directly adopt the reported results from their receptive papers. For fair comparisons, we follow the standard LNL evaluation protocols [Li et al., 2020a; Xia et al., 2019] and adopt the same backbone as in previous works for all benchmarks.

Comparison with synthetic noisy labels. Table 1 shows the results on CIFAR-10/100 dataset with different types and levels of synthetic label noise. ProMix significantly outperforms all the rivals by a notable margin with different settings. Specifically, on the CIFAR-10 dataset with symmetric noise, ProMix exceeds the best baseline by 1.39%, 1.90%, 1.49% and 6.96% under the noise ratio of 20%, 50%, 80%, 90% respectively. Moreover, most baselines exhibit significant performance degradation under heavier noise, whereas ProMix displays robustness and remains competitive. These observations clearly verify the effectiveness of ProMix.

Comparison with real-world noisy labels. As shown in Table 3, on the CIFAR-N dataset with real-world noisy labels, our proposed ProMix achieves the best performance compared with other existing methods on all four settings. In specific, on the CIFAR-10N dataset, the absolute accuracy gains for ProMix are 2.04%, 2.11% and 3.10% under Aggregate, Rand1, and Worst label annotations. Additionally, ProMix further expand the lead and reach a performance gain at 2.66% on the even more challenging CIFAR-100N dataset. Table 2 and Table 4 show that ProMix surpasses all the previous methods on the large-scale Clothing1M and ANIMAL-10N dataset, particularly achieves a lead of 5.88% on ANIMAL-10N. These results clarify that our method can be well applied in real-world scenarios with large-scale data.

4.3 Ablation Study

Effect of sample selection. We first dismiss the label guessing in variant ProMix w/o LGA, where we note that LGA could boost performance, especially under severe noise. Based on this variant, we further equip ProMix with different sample selection strategies: 1) ProMix w/o MHCS which performs CSS and overlooks the remaining matched high confidence samples; 2) ProMix w/o Base Selection which discards the base selection and only selects matched high confidence samples. From Table 5, we can observe that ProMix w/o MHCS largely underperforms ProMix, which verifies the efficacy of MHCS mechanism. ProMix w/o Base Selection cannot establish a competitive performance because of inadequate labeled samples. This also proves that a clean-enough base selection set is the foundation of further progressive selection and debiased SSL. Moreover, We test a variant ProMix with Only Clean which merely employs the selected samples and disregards the unselected samples. This variant suffers from perceptible performance degradation, but still beats all the baselines on CIFAR-N by a large margin. Such result indicates that clean samples are the real treasures in LNL, well-utilizing which is the key to successful learning.

(a) CIFAR-10-Symmetric 80%. (b) CIFAR-100-Symmetric 80%.

Figure 3: Comparison of clean sample selection on CIFAR-10/100 dataset with 80% symmetric noise. The threshold of DivideMix and forget rate of JoCoR have been re-tuned to get the best F1 score.

Effect of bias mitigation. Next, we ablate the contributions of the bias mitigation components in the debiased SSL. Specifically, we compare ProMix with two variants: 1) ProMix w/o Confirmation Bias Removal (CBR) which discards hAP, generating and utilizing the pseudo-labels on the same classification head; 2) ProMix w/o Distribution Bias Removal (DBR) which sticks to the vanilla pseudo-labeling and adopts the standard cross entropy loss. From Table 5, we can observe that the lack of any one of these two components will bring about performance degradation, which indicates the effectiveness of bias mitigation. The performance drop grows more severe in imbalance scenarios as shown in Appendix. Moreover, to reveal the impact of debiasing more intuitively, we compare the accuracy of the pseudo-labels of unlabeled data during training with and without incurring the bias mitigation strategies. As shown in Figure 4, we can see that integrated with the debiasing strategies, the pseudo-labels on unlabeled data grow more accurate, hence facilitating the training process and reciprocating the sample selection process.

ProMix improves the clean sample utility. To further prove that ProMix indeed improves the clean sample utility, we compare the precision and recall of filtered clean samples of ProMix and the other two baselines, DivideMix [Li et al., 2020a] and JoCoR [Wei et al., 2020]. From Figure 3, we observe that ProMix significantly outperforms its competitors in both precision and recall, that is, ProMix selects much more clean samples while introducing fewer noisy samples. In specific, ProMix achieves a precision of 96.59% and a recall of 93.78% on CIFAR-10-Sym 80%, with a lead of 2.23%

(a) CIFAR-10-Symmetric 80%. (b) CIFAR-100-Symmetric 80%.

Figure 4: Accuracy of pseudo-labels for the unlabeled data with and without debiasing on CIFAR-10/100 with 80% Symmetric noise. LGA is disabled in order to avoid unexpected interference.

in precision and 1.15% in recall compared to DivideMix. In more straightforward terms, ProMix picks 450 more clean samples and 891 fewer false positive noise samples than DivideMix, most of which are precious and meaningful hard samples, greatly facilitating training. Besides, on CIFAR100-Sym 80% where performances of competitors decline, ProMix leads by an even wider margin, especially in precision. This indicates that ProMix is based on a pipeline that first selects a clean-enough set with high precision, and then tries to expand the clean set with MHCS while still ensuring high precision, gradually maximizing clean sample utility. See Appendix B for more results of clean sample utility.

5 Conclusion

In this work, we propose a novel framework called ProMix for learning with noisy labels with the principle of maximizing the utility of clean examples. ProMix dynamically expands a balanced base selection set by progressively collecting those samples with matched high confidence for the subsequent SSL. We also incorporate an auxiliary pseudo head and a calibration mechanism to mitigate the confirmation bias and the distribution bias in the process of training. The proposed framework is simple but achieves significantly superior performance to the state-of-the-art LNL techniques on different datasets with various kinds and levels of noise. We hope our work can inspire future research along this direction to promote the LNL methods by well utilizing the clean samples, which are the true treasures of the data.

Acknowledgements

Junbo Zhao wants to thank the support by the NSFC Grants (No. 62206247) and the Fundamental Research Funds for the Central Universities. Lei Feng is supported by the Joint NTUWeBank Research Centre on Fintech (Award No: NWJ-2021005), Nanyang Technological University, Singapore.