Stochastic Feature Averaging for Learning with Long-Tailed Noisy Labels

1 Introduction

Deep neural networks (DNNs) have achieved remarkable success on a wide variety of tasks by leveraging largescale and well-annotated datasets. Nevertheless, data collected from real-world applications usually follow a longtailed class distribution, i.e., most classes are associated with only a small amount of training data, leading to inferior generalization on the minority class [Zhou et al., 2020; Xiang et al., 2020; Menon et al., 2020; Wei and Li, 2020; Cui et al., 2021]. On the other hand, data annotated by human labelers or web crawling are easily corrupted (label noise) in practice. When training with label noise, over-parameterized DNNs can achieve perfect training accuracy due to the memorization effect but cannot generalize [Arpit et al., 2017; Han et al., 2018; Liu et al., 2020; Xia et al., 2021; Wu et al., 2021; Wei et al., 2022b; Zhou, 2022]. Hence, it is crucial to tackle those two types of data biases, i.e., the long-tailed class distribution and label noise, to train robust DNNs.

∗Corresponding author

Many methods for long-tailed learning (LTL) and labelnoise learning (LNL) have been proposed to address those two common types of data biases. LTL aims to deal with the imbalanced class distribution, and LTL methods have been evolving in three main directions: (1) re-balancing the training data [Shen et al., 2016; Liu et al., 2019; Zhou et al., 2020]; (2) adjusting the outputs of a model [Kang et al., 2020; Menon et al., 2020; Tang et al., 2020]; and (3) designing distribution-robust loss functions [Cao et al., 2019; Jamal et al., 2020; Ren et al., 2020]. However, existing LTL methods do not take the presence of label noise into account. On the other hand, LNL aims to deal with the problem of label noise by (1) designing noise-robust loss functions [Ghosh et al., 2017; Liu et al., 2020] or (2) detecting and cleaning the noisy data [Jiang et al., 2018; Li et al., 2020]. As one of the most commonly used criteria, a training sample is suggested as noisy data if the loss value between the prediction and its label is higher than a threshold [Han et al., 2018; Arazo et al., 2019; Xia et al., 2021]. However, these methods can detect many false noisy data because even clean data of the minority class has large losses.

To mitigate such issues for detecting label noise under long-tailed class distribution, we propose a novel and practical framework, stochastic feature averaging (SFA). SFA provides a high-quality splitting of clean and noisy data for both the majority and minority classes. Instead of using the loss values, SFA utilizes the distance distribution between samples and the estimated class centroids in the latent representation space. We find that the class centroid may be inaccurate due to the label noise and data scarcity; thus, we approximate the Gaussian posterior distribution over the running average of estimated class centroids. We sample class centroids from this Gaussian distribution and then compute distances. Based on the selected clean data, we add an auxiliary balance classifier to improve the generalization of the minority class and facilitate the estimate of Gaussian parameters. The proposed SFA framework significantly outperforms various existing approaches by a large margin, illustrated in Figure 1. Further, we propose to combine SFA with different existing methods: the sample-selection approach, distribution-robust, and noise-robust loss functions. The key contributions of this work are summarized as follows:

• We study an under-explored problem of learning from noisy data under long-tailed class distribution, which is more challenging yet practical.
• We propose a novel framework SFA for detecting noisy samples using the distance-based criterion and improving the minority class generalization. SFA can be an alternative to loss-based approaches to boost the performance balance across classes.
• Experimental results show that SFA significantly outperforms existing methods on various datasets and settings. Further, SFA can be used as a universal add-on for mainstream LTL and LNL methods.

2 RelatedWork

Long-tailed learning has drawn significant attention in recent years. Many approaches have been proposed, which can be roughly categorized into three types by modifying: (1) the inputs to a model by re-balancing the training data [Shen et al., 2016; Liu et al., 2019; Zhou et al., 2020]; (2) the outputs of a model, for example by post-hoc adjustment of the classifier [Kang et al., 2020; Menon et al., 2020; Tang et al., 2020] and (3) the internals of a model by modifying the loss function [Cui et al., 2019; Cao et al., 2019; Jamal et al., 2020; Ren et al., 2020]. It is worth noting that most of these LTL approaches rely on the assumption that the labels in the training set are correct. However, this is often not the case in real-world applications and label noise has been shown to deteriorate their performance severely.

Label-noise learning approaches can be broadly divided into three areas of focus: (1) noise transition matrix estimation [Patrini et al., 2017; Hendrycks et al., 2018; Cheng et al., 2022], (2) noise-robust loss functions design [Ghosh et al., 2017; Zhang and Sabuncu, 2018; Liu et al., 2020], and (3) clean sample selection [Han et al., 2018; Jiang et al., 2018; Arazo et al., 2019; Li et al., 2020]. Among these approaches, the "small-loss" criterion, which treats samples with small loss as clean samples, is one of the most frequently used methods and has achieved excellent performance. However, it ignores the class imbalance problem in the training data and thus cannot generalizes to long-tailed datasets.

Learning with long-tailed noisy labels aims to tackle both the long-tailed class distribution and label noise issues. A few attempts have been made in this direction by (1) reweighting samples to put greater emphasis on clean long-tailed data [Ren et al., 2018; Shu et al., 2019; Wei et al., 2022a; Jiang et al., 2022]; (2) designing robust loss functions for handling both label noise and class imbalance [Cao et al., 2020], (3) developing better representation learning methods [Zhou et al., 2022; Yi et al., 2022], and (4) selecting clean samples using carefully designed criteria [Wei et al., 2021; Xia et al., 2021]. Our approach follows the idea of sample selection based on the "small-distance" criterion, which detects noisy samples in the latent representation space. Further, we propose an auxiliary balanced classifier to improve the generalization based on selected clean samples.

3 The Proposed Approach

3.1 ProblemFormulation

Consider a K class classification task, we denote the training dataset D = {(xi, yi)}Ni=1 , where the training sample xi ∈ Rd and its label yi ∈ [K] = {1, 2, ...K}. We group training samples of class k ∈ [K] as Dk = {(xi, yi) | yi = k}. The training dataset D follows a long-tailed class distribution in our problem setting. Specifically, let ρ = maxk |Dk|/mink |Dk| as the imbalance ratio, we have ρ ≫ 1, e.g., ρ = 100. With label noise, a fraction of training samples are incorrectly labeled, i.e., ∃i ∈ [N], yi ̸= y∗ i where y∗ i denotes the ground-truth label for xi. In the rest of the paper, we denote the fraction of incorrectly labeled training samples as γ ∈ (0, 1). Given D, our goal is to learn a classifier f : Rd → [K] that can generalize to unseen data.

Basic idea. To tackle the problem of learning from longtailed noisy data, we design a novel framework termed SFA. On the one hand, SFA follows the sample selection paradigm and selects clean samples in the latent representation space, which applies to both the majority and minority classes. On the other hand, we propose to employ an auxiliary balanced classifier for training using selected yet long-tailed clean samples to improve the generalization for the minority class. Figure 2 provides an overview of our proposed framework.

3.2 ANew Sample Selection Framework

The Small-Distance Criterion. To select clean samples from long-tailed and noisy data, we proposed to adopt the small-distance criterion. It has been verified that the smalldistance criterion is more effective than the small-loss counterpart in the recent literature [Wei et al., 2021]. The smalldistance criterion makes an assumption on the data distribution that the likelihood of a sample xi belonging to class k decays exponentially with its distance from its class centroid ck, i.e., P(xi | ck) ∝ e-dist(ck,xi) , where dist is the distance measure in the latent representation space and is typically set to be the Euclidean distance. The assumption allows for an unrestricted number of samples, making it suitable for both the majority and minority classes. In other words, the closer the samples are to their corresponding class centroids, the more likely they are to be clean samples. The challenge of this approach is how to accurately estimate the class centroids in the presence of label noise, even when only a few samples are accessible for the minority class. In the following, this paper seeks to solve this problem by estimating the posterior distribution of the class centroids.

Instant Centroid Estimation. A direct approach to estimate the class centroids is to compute the mean of all feature vectors of each class after each training epoch referred to as the instant centroid estimation. However, it may be difficult to produce accurate estimates when label noise exists in data, as it uses noisy samples in the computation. To mitigate this issue, one feasible approach is to use as many clean samples as possible to compute the class centroids. We find that the majority and minority classes tend to have similar ranges of predicted confidence output by the auxiliary balanced classifier, and most clean samples have higher confidence than noisy samples. This observation motivates us to select a fraction of clean samples for each class to compute a rough estimate of class centroids using a single confidence threshold τ . Specifically, we compute the centroid of class k by:

where I(·) is the indicator function, N ′k is the number of selected samples, g(x) returns the latent representation for x, and fabc(x) outputs the predicted confidence by the auxiliary balanced classifier, which is introduced in Section 3.3. In practice, since more and more samples tend to receive high predicted confidence as the training proceeds, we set a dynamic threshold τt as an increasing function of iteration t, which is given by τt = ϕt ˆ τ, where ϕ is a constant. For example, we set ϕ = 1.005 and ˆ τ = 1/K in our experiments. With this instant centroid estimation, we obtain a relatively reliable centroid for selecting clean samples. However, the accuracy of the estimate can still be affected by the scarcity of the minority class samples.

Stochastic Feature Averaging. To further reduce the estimation error, inspired by [Maddox et al., 2019], a Bayesian extension of the class centroid is designed to conduct Bayesian inference using a Gaussian approximation to the posterior distribution over class centroids. To start with, SFA maintains the running average of class centroids:

where CSFA = [c1; · · · ; cK] is the running estimate of class centroids, Ct denotes the instant centroid estimates computed by Eq. (1) after the t-th training epoch, and β is the smoothing factor. This averaging in the latent representation space captures the training dynamics of DNNs and provides a robust estimate of the centroid.

SFA conducts Bayesian inference using Gaussian approximation to the posterior distribution over class centroids. This paper considers a simple diagonal format for the covariance matrix. To fit a diagonal covariance approximation, we maintain a running average of the second moment for class centroids:

C2 denotes the element-wise square. Thus, a diagonal covariance matrix can be approximated by ΣSFA = diag(C′SFA - C2SFA) and the posterior over class centroid can be constructed as a Gaussian distribution N(CSFA,ΣSFA). Note that previous work [Maddox et al., 2019] also proposes a higher-rank approximation for the covariance, but we only focus on the diagonal approximation for simplicity. Then we sample several stochastic class centroids from this Gaussian distribution and perform distance averaging for sample selection. Specifically, given the sampling rates S, we first sample ˜C ∼ N(CSFA,ΣSFA) and then compute the Euclidean distances between ˆck and samples of class k by:

This procedure is repeated for S times and an averaged distance is obtained; thus, training samples are well clustered with the distance to the class centroid by fitting them into a two-component Gaussian mixture model (GMM) [Permuter et al., 2006], i.e., dist ∼ ∑2j=1 ϕjN(µj, σ2 j ) where µj , σ2 j

Algorithm 1: The SFA Framework

1 Input: training dataset D = {(xi, yi)Ni=1 }, model parameters θ, sampling rate S, warm-up epochs T0, and total training epochs T . // warm-up for T0 epochs 2 for t = 1, ..., T0 do 3 L = ℓCE(D, f) + ℓBS(D, fabc) 4 θt = SGD(L, θt-1) 5 end 6 for t = T0 +1, ..., T do // sample selection 7 Dclean = ∅, Dnoisy = ∅ 8 Compute confidence threshold τt = γ t ˆ τ 9 for k = 1, ..., K do 10 Compute instant centroid by Eq. (1) 11 Update the first moment CSFA by Eq. (2) and the second moment C′SFA by Eq. (3) 12 for s = 1, ..., S do 13 Sample ˜ck ∼ N (CSFA, ΣSFA) 14 disti,s = ||˜ck - g(xi)||2, where xi ∈ Dk 15 end 16 disti = 1S ∑Ss=1 disti,s 17 Dclean k ,Dnoisy k = GMM(dist) 18 Dclean = Dclean ∪Dclean k ,Dnoisy = Dnoisy ∪Dnoisy k 19 end // semi-supervised learning 20 LSSL =MixMatch(Dclean ,Dnoisy , f) 21 LABC =MixMatch(Dclean ,Dnoisy , fabc) 22 L = LSSL +LABC 23 θt = SGD(L, θt-1) 24 end

are the mean and variance of the j-th Gaussian component. Assume µ1 < µ2 without loss of generality, as clean samples distribute around class centroids while noisy samples spread out, we denote the clean probability of one sample as:

We select samples with P(clean | xi) > 0.5 as clean samples and others as noisy samples. Next, the training dataset D is divided into the clean sample set Dclean and noisy sample set Dnoisy . As data in Dclean may still follow a long-tailed distribution, we propose an auxiliary balanced classifier to obtain unbiased predictions.

3.3 Auxiliary Balanced Classifier

In the literature on long-tailed learning, class re-balancing strategies are the prominent and effective methods proposed to alleviate imbalance problems, which can significantly promote classifier learning and affect the representation learning w.r.t. the original data distribution. Several existing works [Kang et al., 2020; Zhou et al., 2020; Lee et al., 2021] suggest decoupling representation and classifier learning and demonstrate their superiority over conventional learning methods.

To further boost the performance, we adopt a two-branch network to combat class imbalance during model training: we add an auxiliary classifier to the backbone of the neural network for balanced classifier learning and meanwhile maintain the original classifier for the sake of representation learning. We denote the two classifiers as fabc and f, where f is trained using the standard cross-entropy loss ℓCE(x, y) = -log ezy ∑Kk=1 ezk , where z = f(x). Next, we describe obtaining a balanced classifier fabc.

Balanced Softmax. To produce unbiased predictions, we propose an auxiliary balanced classifier fabc, which is jointly learned with the standard classifier f by sharing the feature extractor. Specifically, fabc seeks to minimize the Balanced Softmax (BS) function [Ren et al., 2020]:

where nk = |Dclean k | is the number of samples of class k counted from Dclean , and zk stands for the k-th logit produced by fabc(x). The loss can penalize more heavily on samples in the majority class while placing a lower penalty on samples in the minority class. This helps to learn a balanced classifier and prevent the model from being significantly biased towards the majority class.

Incorporating with Semi-supervised Learning. To improve the utilization of noisy data, we treat samples in noisy set Dnoisy as unlabeled data and incorporate our methods into a popular semi-supervised learning framework MixMatch [Berthelot et al., 2019]. MixMatch is a state-of-the-art semisupervised learning algorithm that combines data augmentation, consistency regularization, and mixup to achieve excellent performance. For unlabeled samples in Dnoisy , we perform label guessing using predictions from the standard classifier f. The total loss is L = LSSL +LABC, where LSSL and LABC stands for a MixMatch loss computed from the standard classifier f and the balanced classifier fabc. The pseudo-code of our SFA framework is summarized in Algorithm 1. At test time, both classifiers are utilized to return the classification results. Furthermore, we also employ two separate neural networks to combat confirmation biases following DivideMix [Li et al., 2020] and RoLT+ [Wei et al., 2021].

3.4 Time Complexity Analysis

The computation of distances between features and class centroids, and the fitting of GMM, are the main additional computational cost in sample selection. The time complexity of computing distances is O(SNd), where N is the number of samples, d is the latent feature dimension, and S is the sampling rate. We set S = 1 for large-scale datasets. The time complexity of fitting the two-component GMM using the EM algorithm requires O(Nd′ ct), where d′ is the dimension of distances, c is the number of components, and t is the number of iterations. In our approach, we have d′ = 1, c = 2, and t = 100. The total time complexity is O((200+d)N), which is efficient enough compared with existing approaches.

4 Experiments

4.1 Datasets and Implementation Details

Benchmark Datasets. We first test our approach on CIFAR-10 and CIFAR-100 datasets by simulating training data with long-tailed class distribution and label noise following prior work [Wei et al., 2021]. Formally, denote the imbalance ratio as ρ and noise level as γ. We set the number of samples for the k-th class to Nk = N/ρ k-1 K-1 and generate a long-tailed dataset. Next, we inject the label noise into this dataset via a noise transition matrix T defined as:

We use an 18-layer PreAct ResNet [He et al., 2016] and train it using SGD with a momentum of 0.9, a weight decay of 5× 10-4 , a batch size of 128 and an initial learning rate of 0.02. The model is trained for 200 epochs with 1 NVIDIA GeForce RTX 3090. We perform sample selection after a warm-up period of 30 epochs and anneal the learning rate by a factor of 10 after 150 epochs. For all CIFAR experiments, we choose ρ from {10, 50, 100} and γ from {0.2, 0.5}, and use the same hyperparameters β = 0.99 and S = 5.

Real-World Dataset. WebVision is a large-scale dataset with real-world noisy labels and long-tailed distributions. It contains 2.4 million images crawled from Flickr and Google using the 1,000 concepts in ImageNet ILSVRC12. As previously done in [Li et al., 2020], we use the Inception-ResNet v2 architecture [Szegedy et al., 2017] to evaluate the performance of baseline methods on the first 50 classes of the Google image subset. We set the hyperparameters to β = 0.9, S = 1 and train the network using SGD with a momentum of 0.9, a weight decay of 1 × 10-3 , and a batch size of 32. The initial learning rate is set to 0.01 and reduced by a factor of 10 after 50 epochs. The warm-up period is one epoch, and the model is trained for 100 epochs in total with 2 NVIDIA GeForce RTX 3090.

4.2 ComparisonMethods

We compare the performance of our approach SFA with nine baselines with the same network architecture. In addition to the direct approach using cross-entropy loss (CE), we compare SFA with the following three groups of methods:

• Long-tailed learning. Methods commonly used to address the long-tailed classification include BBN [Zhou et al., 2020] and cRT [Kang et al., 2020].
• Label-noise learning. Recent state-of-the-art methods for learning with noisy labels, including ELR+ [Liu et al., 2020] and DivideMix [Li et al., 2020]
• Learning with long-tailed noisy data. Methods designed to tackle noisy labels and long-tailed distributions simultaneously, including MW-Net [Shu et al., 2019], HAR[Cao et al., 2020], RoLT+ [Wei et al., 2021], and PCL [Wei et al., 2022a].

4.3 Results on Benchmark Dataset

Table 1 summarizes the results on CIFAR-10 and CIFAR-100 datasets under different levels of label noise and imbalance ratios. We report the "best" test accuracy across all training iterations and the "last" test accuracy at the end of training. As shown in the results, initial model training with crossentropy loss is susceptible to biases in the training set, leading to a significant decrease in its classification accuracy as the data biases become more pronounced. Also, previous longtailed learning methods (i.e., BBN and cRT) dreadfully degrade their performance as the noise level increases. When the noise level is high, the last accuracy is significantly lower than the best accuracy due to the model's tendency to fit the label noise, which can negatively affect the model's ability to generalize to unseen data. The same conclusions can be drawn from other methods. However, our approach (SFA) retains the most robust performance and advances all other compared methods in almost all dataset settings. In particular, compared with the previous state-of-the-art method PCL, SFA can further improve the test accuracy on CIFAR-10 by an average of 4.04% and on CIFAR-100 by an average of 1.24%, and it can be observed that the improvement becomes more significant at high noise levels and imbalance ratios, benefiting from proposed stochastic feature averaging and balanced classifier training.

4.4 Results on Real-World Dataset

Table 2 reports the results on the WebVision dataset (ρ ≈ 6). SFA consistently outperforms compared methods by a large margin, with an average improvement of 1.02% (0.9%) on WebVision (ImageNet) validation sets. This demonstrates the effectiveness of our approach in improving the performance of deep learning models in the presence of real-world label noise and class imbalance. To further illustrate the benefits of our approach, we also conduct experiments on the WebVision dataset by manipulating the imbalance ratios (ρ = 50 and ρ = 100) and observe a clear improvement, which suggests that the advantage of our approach is more appealing as the imbalance factor increases.

4.5 Ablation Studies and Further Analyses

Insights into the Key Components. We conduct ablation studies to better understand the impact of different components of our SFA framework. Table 3 reports the results on CIFAR datasets with varying levels of noise and imbalance ratios, where w/o ICE/SCC/ABC means removing the proposed instant centroid estimation, stochastic class centroid, and auxiliary balanced classifier, respectively. The following analyses of results can be derived: (1) in the presence of noisy labels, instant centroid estimation based on the confidence of samples from the balanced classifier can provide a more accurate approximation of the actual class centroids compared to simply taking the mean value of all sample features. (2) The dramatic decrease in accuracy among all settings without stochastic class centroid highlights the significance of the Gaussian posterior distributions over class centroid for better sample selection. (3) the auxiliary balanced classifier can significantly enhance the model's performance, especially in cases of heavy class imbalance.

Efficacy of Sample Selection. We compare the F1-scores with existing sample selection-based methods (i.e., DivideMix and RoLT+) on two CIFAR datasets to verify the effectiveness of sample selection in our approach. Results in Figure 4 illustrate the superiority of our approach in selecting clean samples from noisy and long-tailed data. We believe this is due to the following reasons: (1) we adopt the smalldistance criterion, which is more robust than small-loss when noisy labels and class imbalance co-exist in datasets; (2) sampling stochastic class centroid from the Gaussian posterior distribution, which is used in distance averaging, can further reduce the impact of label noise and data scarcity on the estimation of the actual class centroid; (3) conducting Bayesian inference of class centroid allows the model to access a diverse range of clean samples at different epochs, which helps to prevent overfitting to a particular clean subset. This can be further illustrated in Figure 5. We compare the performance of our method with RoLT+ [Wei et al., 2021] in terms of the mean loss of actual clean samples on CIFAR-10 under different data settings. Both methods use the small-distance criterion, but our method utilizes Bayesian inference to estimate the class centroids. We believe a method that can achieve lower losses on the whole clean samples can more effectively identify and use these samples to improve the performance. The plot demonstrates that our approach can obtain lower losses and, therefore, better learning from clean samples than the method without stochastic class centroids.

Collaboration with Existing Approaches. To further showcase the versatility of the proposed sample selection approach, we present its three applications. We combine SFA with a variety of existing techniques, including the sample-selection approach (Co-teaching [Han et al., 2018]), the distribution-robust loss (LDAM [Cao et al., 2019]), and the noise-robust loss (HAR [Cao et al., 2020]). Specifically, we use SFA in two ways: a replacement for the sample selection module in Co-teaching and a method for identifying clean samples to train models with LDAM and HAR. Figure 3 summarizes the performance of different approaches with and without applying SFA under various settings. Results show that SFA improves the generalization of all approaches, leading to a more robust model.

5 Conclusion

This paper proposes a distance-based sample selection approach called stochastic feature averaging (SFA) and an auxiliary balanced classifier for handling datasets with both noisy labels and long-tailed class distribution. The SFA framework first estimates an instant centroid for each class and uses the exponential running average of the centroid to fit a Gaussian posterior distribution, which is utilized to identify noisy samples based on their distances to the centroid sampled from the Gaussian. Next, a standard and an auxiliary balanced classifier are jointly learned via a semi-supervised learning framework to improve the generalization of the minority class and facilitate the estimation of the Gaussian parameters. The proposed approach is evaluated on various datasets, including CIFAR-10, CIFAR-100, and WebVision, and is shown to outperform previous state-of-the-art consistently. Our proposed approach, SFA, can be an alternative to existing loss-based approaches for learning with long-tailed noisy labels and can be used as a universal add-on for various methods. For future work, we plan to investigate and provide a theoretical explanation for the effectiveness of stochastic class centroids.

Acknowledgments

This research was supported by the National Science Foundation of China (62206049). This research work is supported by the Big Data Computing Center of Southeast University.