Fine-tuning Pre-trained Models for Robustness under Noisy Labels

1 Introduction

Despite the remarkable performance of deep neural networks (DNNs), their effectiveness decreases significantly when trained with inaccurate supervision. Additionally, manually correcting noisy labels or acquiring clean labels anew is challenging due to the large-scale nature of datasets. To address this issue, researchers have developed various approaches within the field of Learning with Noisy Labels (LNL). These approaches encompass robust training loss [Zhang and Sabuncu, 2018; Wang et al., 2019], regularization [Cheng et al., 2023; Ko et al., 2022], and semi-supervised learning methods [Li et al., 2020; Liu et al., 2020].

However, recent studies in the field of LNL have shown an increase in computational complexity. As demonstrated in Table 1, one of the notable algorithms called UNICON [Karim et al., 2022] incurs a substantial training time increase of 622% compared to the vanilla approach. This is primarily due to the utilization of noisy labeled samples. Notably, the most computationally expensive aspect is the careful detection and integration of noisy labeled samples. For example, UNICON employs an expensive contrastive loss, resulting in higher computational costs. The motivation behind such a costly incorporation of noisy labels is to enhance the generalization performance, which can be derived from a larger training dataset.

We are motivated by the conjecture that pre-trained models (PTMs), widely recognized for their strong generalization performance with fast adaptation, have the potential to effectively address the limitations of previous denoising algorithms. Recent research supports this intuition by demonstrating that PTMs can quickly adapt to new target datasets using few-shot correctly labeled samples and achieve generalization performance across a wide range of tasks [He et al., 2022; Assran et al., 2022]. This ability is attributed to their robust feature extraction capabilities. Therefore, the use of PTMs in LNL methods can contribute to improving generalization performance in a few epochs while simultaneously reducing computational costs by minimizing the involvement of potentially noisy labels.

However, there has been limited research on the application of PTMs to noisy labeled datasets and the effective utilization of their valuable knowledge in such scenarios. Therefore, there is a need to investigate methods that are both efficient and effective in leveraging the robust feature extractor of PTMs in the presence of noisy labels.

This research introduces a robust method to transfer the knowledge of PTMs to a target dataset that probably has noisy labels. To this aim, two adaptation methods are explored: full fine-tuning (FFT) and linear probing (LP). In the FFT approach, all parameters of the PTM are updated, while in the LP tune only the last fully-connected (FC) layer with frozen feature extractor.

Contribution. The main observations and contributions of this research are summarized as follows:

• It is confirmed that when a high (low) proportion of noisy

labels is present, the feature extractor becomes distorted (empowered) when tuned using FFT, respectively. Since we lack sufficient information on the noisy ratio, developing a proper algorithm is needed. 1

• The proposed method, fine-TUning pre-trained models for Robustness under Noisy labels (TURN), consists of two adaptation steps (as described in Figure 1): LP and then FFT. The LP step aims to adapt the model to the target task without compromising the integrity of the feature extractor, enabling the effective detection of noisy samples. The FFT step further enhances the feature extractor by updating entire trainable parameters, followed by dataset cleansing.
• Experimental results demonstrate the efficiency and robustness of the proposed method compared to existing LNL methods on various datasets, including synthetically noisylabeled datasets like CIFAR-100, as well as real-world datasets such as WebVision and Clothing1M.

2 Preliminary

Learningwith noisy labels. Let us denote the training dataset as Dtrain = ¶(xi↪ ¯ yi)♦Ni=1 , comprising N pairs of input images xi and their corresponding labels ¯ yi 2 ¶1↪ ▷ ▷ ▷ ↪C♦. In realworld scenarios, the given label ¯ yi can be corrupted due to various factors, such as human errors in crowd-sourced labeling systems. We use yi to denote the ground truth label for (xi↪ ¯ yi), which is not accessible during the training phase. It is widely acknowledged that models trained on noisy labeled datasets Dtrain using conventional classification loss functions, such as cross-entropy loss (LCE), often exhibit poor performance on test data. This limitation poses challenges when deploying such models in real-world scenarios. Therefore, training robust models capable of effectively handling noisy labeled datasets becomes essential.

1 A similar observation was noted in a previous study [Cheng et al., 2023], but their findings were only focused on a SimCLR model trained on the target dataset. However, we expand this to the models trained on different datasets (e.g., ImageNet) and various PTMs.

Pre-trained model and fine-tuning. In recent times, several PTMs have been proposed for image-related tasks, encompassing models trained using supervised learning (SL) [Dosovitskiy et al., 2021; Liu et al., 2021; Liu et al., 2022], selfsupervised learning (SSL) [He et al., 2022; Assran et al., 2022], and multimodal learning [Radford et al., 2021]. They have become accessible and demonstrated significant performance in classification tasks. To harness the power of the pre-trained feature extractor, two primary tuning methods are commonly employed: linear probing (LP) and full fine-tuning (FFT). We represent the classification model with a PTM as g(f(x; ✓); φ), where g(·; φ) denotes the linear classifier with its parameter φ, and the feature extractor f(·; ✓) has its parameter ✓. In LP, ✓ remains frozen and only φ is trained. In contrast, FFT involves tuning all trainable parameters, including both ✓ and φ. The advantages and disadvantages of LP and FFT are fundamentally distinct. LP has the strength of inexpensive computational cost thanks to the limited number of trainable parameters, but has rigid adaptability. On the other hand, FFT has strong adaptability, but requires a large amount of training resources.

3 Motivating Observation

In this section, our objective is to analyze the behavior of the feature extractor when tuned on severely or slightly noisy labels. We particularly focus on identifying the conditions that lead to improvements in the feature extractor. The key observations that form the basis of our investigation are summarized as follows.

(Obs 1) The presence of a high proportion of noisy labels can significantly distort the feature extractor when FFT is applied. (Obs 2) Conversely, when the noise ratio is not severe, FFT can effectively enhance the feature extractor, allowing it to construct class-wise clusters accurately.

Inspired by these observations, we propose a denoising algorithm for PTMs, which is elaborated in Section 4. In subsequent paragraphs, we provide detailed explanations for these two motivations. 2

Experimental Setting. In our investigation, we performed experiments to evaluate the performance of several popular PTMs on a dataset with noisy labels. The considered PTMs include ViT-B/16 [Dosovitskiy et al., 2021], ConvNeXt-T [Liu et al., 2022], CLIP-ViT-B [Radford et al., 2021], and MAEViT-B [He et al., 2022]. To generate the noisy labeled dataset, we employ the symmetric label-flipping technique on the CIFAR-100 dataset. This technique randomly flips the labels of either 90% or 10% of the samples, resulting in different levels of label noise. We investigate the impact of noisy labels by visualizing the embedding of randomly sampled 10 classes from the test dataset using t-SNE [Van der Maaten and Hinton, 2008] plots. Each model is trained for 5 epochs following [Kumar et al., 2022].

(Obs 1) In the severe noise case, applying FFT results in distortion of the feature extractor. The distortion of the feature extractor occurs due to the incorrect supervision signals by the noisy labels. To gain a deeper understanding of this phenomenon, we employ t-SNE plots. In Figure 2, the middle row illustrates the initial feature extractor of each PTM, while the bottom row displays the features after applying FFT with 90% noisy labels. It can be seen that the features become mixed and less distinguishable. This effect is particularly evident when the initial clusters of the PTMs are already poorly defined, i.e., MAE. As a result, the application of FFT on a severely noisy dataset leads to a trained feature extractor that fails to adequately capture the intrinsic characteristics of the samples.

2 There was an observation of the impact of FFT and LP for PTMs [Kumar et al., 2022]. Its main focus is on scenarios where PTMs are fine-tuned on Dtrain ⇠ (X↪ Y) and tested on out-ofdistribution dataset, i.e., Dtest ⇠ (X 0 ↪ Y), where X 0 6= X . In contrast, we mainly focus on noisy label case, where the input images x in the training and testing datasets follow identical distribution while the labels in the training dataset is inaccurate.

(Obs 2) FFT improves the feature extractor when the target dataset has slight label noise. When dealing with minor label noise in the target dataset, applying FFT to the feature extractor yields improvements in constructing class-wise clusters effectively. The upper row of Figure 2 reveals that the models under examination exhibit a better construction of class-wise clusters compared to the middle row. This observation suggests that fine-tuning the feature extractor on a dataset with slight label noise can enhance its performance, enabling it to capture more meaningful features from each sample.

4 ProposedMethod: TURN

Based on the findings in Section 3, we can extract crucial strategies for the effective use of PTMs in noisy label datasets. First, if the training dataset is significantly contaminated with label noise (i.e., Obs 1), it is critical to avoid directly incorporating these noisy labels, as they can negatively impact the valuable feature extractor of PTMs. Hence, a buffer, such as LP, is needed to manage these highly noisy instances, as it allows the feature extractor to remain frozen. Second, when the training dataset exhibits relatively low label noise (i.e., Obs 2), FFT should be employed for task adaptation. This is due to the need for the feature extractor to be adjusted to the target dataset after FFT in the presence of minor noise. However, a key challenge remains: the lack of specific information on the noise label, including the intensity of the noise ratio. Therefore, the suggested approach to protect the feature extractor is a two-step procedure involving LP, followed by FFT. This section introduces our proposed two-step method, referred to as TURN (fine-TUning pre-trained models for Robustness under Noisy labels), aimed at addressing these challenges.

Algorithm 1 Pseudo code of TURN

Input: Dataset Dtrain = ¶(xi↪ ¯ yi)♦Ni=1 , Linear classifiers g(z; φ), Pre-trained feature extractor f(x; ✓), GMM threshold ⌧ , Linear probing epoch ELP, Full fine tuning epoch EFFT Output: Fine-tuned model g(f(·)) /* Step 1: Linear probing */ Extract feature zi = f(xi; ✓) for all samples (xi↪ ¯ yi) 2 Dtrain for e < ELP do Train the linear classifier g(zi; φ) under GCE Loss: LGCE(g(zi; φ)↪ ¯ yi) end /* Step 2: Select clean samples and FFT */ for e < EFFT do Extract per-sample loss for all samples in Dtrain Construct clean subset Dclean by using GMM with threshold ⌧ Train the model g(f(xj ; ✓); φ) under LCE(g(f(xj ; ✓)φ)↪ ¯ yj ) on Dclean = ¶(xj ↪ ¯ yj )♦

end

4.1 AlgorithmDescription

In this section, we provide a detailed explanation of the proposed algorithm. Briefly speaking, The algorithm consists of two main steps. In Step 1, the algorithm utilizes LP with an initialized fully connected layer, denoted as g(z; φ), where z represents the output of the frozen feature extractor f(x; ✓) for each input image x. Subsequently, in Step 2, the algorithm iteratively cleanses the training dataset and performs FFT on the entire model g(f(x; ✓); φ) using a subsampled dataset Dclean. The procedure of TURN is described in Algorithm 1.

Step 1: Linear probing. The first step of the algorithm serves two main objectives. Firstly, the objective is to obtain a classifier that can detect noisy labels, which is consistent with the approach taken by previous works that utilize a warm-up phase before detecting noisy labels [Li et al., 2020; Karim et al., 2022; Kim et al., 2021]. Second, during the training of the classifier for detecting noisy labels, it is crucial to ensure that the feature extractor is not distorted by noisy labels. To address this, we employ LP, which helps protect the feature extractor from being affected by severe label noise. As mentioned in Section 3, there is evidence that the feature extractor can be improved through FFT when applied to datasets with a small proportion of noisy labels. However, since the exact degree of label noise is unknown, we adopt a conservative approach by freezing the feature extractor until we have a classifier capable of detecting noisy labels.

Additionally, applying LP enhances efficiency, which is a key consideration derived from the inherent strengths of LP, as described in Section 2. Therefore, we employ the following implementation technique. Since the feature extractor remains frozen in this step, the extracted features of the input images remain unchanged. As a result, we pre-extract the features zi of input images xi as zi = f(xi; ✓) in advance, minimizing computational overhead during the subsequent steps.

Next, we update the parameters φ for ELP epochs by utilizing the Generalized Cross Entropy loss [Zhang and Sabuncu, 2018], denoted as LGCE = 1-g(zi;φ)q q , where q is a hyperparameter. GCE loss is used to mitigate the influence of noisy labels during linear classifier training. This step is crucial for effectively training the linear classifier, as it helps to clean the dataset in preparation for the subsequent steps.

Step 2: Cleansing and FFT In Step 2, the main objective is to improve the performance of the model by adapting it to the target dataset. As mentioned in Section 3, applying FFT to a dataset with a slight noise label enables the feature extractor to better adapt to the target dataset. Therefore, the remaining part of this step focuses on obtaining a sufficiently cleansed dataset. This stage is divided into two substeps: cleansing and FFT. In the cleansing phase, clean samples are selected from the given noisy dataset, while discarding the remaining samples. Following the cleansing phase, FFT is carried out on the selected clean subset, to improve the performance of the model on the target dataset. These two substeps are carried out for EFFT epochs iteratively. This is because a model that is trained better exhibits a greater ability to distinguish noisy labels.

Note that we can achieve efficiency by discarding the noisy labeled samples during the FFT procedure, leveraging the inherent strength of PTMs. As mentioned above, PTMs are highly capable of adapting to clean datasets using few-shot learning techniques. This allows the model to achieve successful performance even with a small portion of clean training samples. In the subsequent sections, we will provide a comprehensive description of each substep involved in the FFT process.

◦ Step 2-1: Selection of clean samples. To identify clean samples, we utilize a clustering algorithm based on the Gaussian Mixture Model (GMM), a commonly employed in previous research works [Li et al., 2020; Kim et al., 2021; Ahn and Yun, 2023]. This algorithm begins by calculating the loss for each sample, which is determined based on its noisy labels. This process allows us to distinguish between clean and noisy samples in the dataset.

After obtaining the per-sample loss, the next step involves fitting the GMM model using the calculated losses ¶`i♦Ni=1 . From this GMM model, we extract two Gaussian distributions for each class c, i.e., pcl and pch . Here, pcl represents the distribution with a lower mean value compared to pch . Utilizing these per-class distributions, we construct the clean dataset using the following procedure,

where Dc clean = ¶(xi↪ ¯ yi)♣pcl (`i) > ⌧ where ¯ yi = c♦. In this formulation, the threshold hyperparameter ⌧ is used, and the function U(D↪ n) denotes the uniform sampling function, which randomly selects n samples uniformly from the set D. It is worth noting that an equal number of samples is selected for each class from the set considered clean. This selection strategy is based on the principle emphasized in [Karim et al., 2022], which highlights the importance of maintaining a uniform distribution. Therefore, we set n = minc2¶1↪▷▷▷↪C♦ ♣Dc clean ♣↪ where ♣D♣ represents the cardinality of the set D.

◦ Step 2-2: FFT on the selected dataset Dclean. In Step 2-1, a clean subset is extracted from the noisy target dataset. This subset is obtained by reducing the proportion of noisy labels, thereby creating an environment conducive to performing FFT. With a reduced influence of noisy labels, the FFT process can be executed, allowing the model to adapt more effectively to the underlying clean samples present in the training dataset.

5 Experiments

In this section, we present empirical evaluations that showcase the superior performance of TURN. We begin by providing a detailed description of the LNL benchmarks and implementation in Section 5.1. Subsequently, in Section 5.2, we present the experimental results obtained from extensive evaluations. Further analyses to gain a deeper understanding of TURN are described in Appendix.

5.1 Experimental Setting

Datasets. We conduct an evaluation on the synthetically noised CIFAR-100 dataset, and real-world noisy labeled dataset, the Clothing 1M and WebVision datasets. For the CIFAR dataset, we introduce uniform random noise into a portion of the labels to simulate symmetric noise, asymmetric noise which flips the labels to specific classes [Liu et al., 2020]. To incorporate instance-dependent noise, we adopt the noise generation methodology described in [Cheng et al., 2021].

Architectures and baselines. In our evaluation, we consider several PTMs for each dataset, including ViT-B/16 [Dosovitskiy et al., 2021], ConvNeXt-T [Liu et al., 2022], MAE-ViTB [He et al., 2022], MSN-ViT-B [Assran et al., 2022], CLIPViT-B [Radford et al., 2021], and ResNet-50 [He et al., 2016]. In the case of CLIP model, we utilize the visual model. Among these architectures, we compare our proposed TURN with various previous methods, including Vanilla (trained on crossentropy loss), GCE [Zhang and Sabuncu, 2018], ELR [Liu et al., 2020], DivideMix [Li et al., 2020], and UNICON [Karim et al., 2022]. Additionally, we apply both FFT and LP to all algorithms for comparison. However, it is worth noting that DivideMix and UNICON require a significant number of feed-forwards for each sample, approximately 4 times and 8 times, respectively, with various data augmentation techniques. Due to the computational limitation, we utilize LP for both DivideMix and UNICON. For further details, please refer to Appendix B.

Implementation. To optimize the hyperparameters for each model, we utilize the Ray [Liaw et al., 2018] hyperparameter tuning tool. This allows us to identify the appropriate settings for parameters such as learning rate, weight decay, optimizer, and batch size. For each PTM, specific optimized hyperparameters and their search spaces are described in the Appendix A. Regarding the hyperparameter for TURN, namely GMM threshold, ⌧ = 0▷6, we adopt the value suggested in the DivideMix paper [Li et al., 2020], which initially introduced the GMM-based clean sample selection. For baselines, we run 5 epochs for FFT and 20 epochs for LP, while TURN is optimized 20 epochs of LP with 4 epochs for FFT in Step 2 to spend smaller computational cost compared to the baselines.

5.2 Classification Result

CIFAR datasets. For CIFAR-100 dataset, we include four noisy label cases {Symm 0▷6, Symm 0▷9, Asym 0▷4, Inst 0▷4}. The results shown in Table 2 demonstrate that TURN, when applied to various PTMs, consistently delivers high performance across different types and severities of noisy labels. Furthermore, the proposed method exhibits the superiority of the FFT and LP tuning mechanisms. Specifically, the ViT-B/16 model consistently outperforms others. As discussed in Section 3, in cases of severe label noise (i.e., Symm 90% noise), the LP-based approach generally yields more stable performance compared to FFT. Conversely, in cases of less severe label noise (e.g., Symm 60% noise), FFT tends to outperform LP. This tendency can be understood by Obs 1 that FFT under severe noise can distort the feature extraction. However, TURN approach consistently outperforms both cases. Therefore, it can be concluded that the sequential combination of LP followed by FFT is a valuable way for tuning PTMs on noisy datasets.

Real-world tasks. We evaluate the performance of TURN on larger datasets, namely Clothing1M and WebVision. The results in Table 3 consistently demonstrate that TURN improves performance on these datasets. This highlights the ability of TURN to handle noisy labels in real-world scenarios. However, some models, like MAE, MSN, and ResNet on Clothing1M, show a drop in performance due to the fine-grained nature compared to the training dataset (i.e., ImageNet). Nonetheless, our algorithm performs better by utilizing FFT in the second step.

5.3 Further Analysis

We deliver further analyses to answer the following keywords: (1) efficiency, (2) hyperparameter sensitivity (3) larger model case, and (4) ablation study. They are included in Appendix.

Larger model analysis. We also examine the performance of the proposed algorithm using larger models (ViT-L/16 and CLIP-ViT-L/14) on the CIFAR-100 dataset with 90% symmetric case. As shown in Table 4, the proposed algorithm consistently outperforms the other FFT and LP cases. Notably, when using larger models, the performance tends to improve compared to the results in Table 2. However, it is important to consider that larger models require more computation resources and time for tuning due to the increased number of model parameters.

Hyperparameter sensitivitiy. In the proposed method, we analyze three types of hyperparameters: (1) the number of epochs for LP denoted as ELP, (2) the number of epochs for FFT denoted as EFFT, and (3) the GMM threshold ⌧. Our experiments on CIFAR-100 with 90% symmetric noise reveal that increasing the number of epochs for LP improves performance until reaching 20 epochs, beyond which the gain becomes negligible. Similarly increasing the number of FFT epochs up to 5 provides similar performance due to the similar sample size per epoch. Moreover, increasing the GMM

threshold results in a correct subset of improved performance. Therefore, careful tuning of these hyperparameters can contribute to enhanced performance.

Component analysis. To evaluate the impact of each component of TURN, we conducted experiments on the CIFAR-100 dataset under 90% symmetric case. Specifically, we examined three components: (1) LP, (2) Cleansing, and (3) FFT. Regarding Cleansing, we explored different types: None (no cleansing), Once (cleansing once right before FFT), and Multiple (cleansing at the beginning of Step 2). The results, as shown in Table 5, indicate that the configuration involving all components achieves the best performance. Additionally, LP acts as a buffer to prevent the feature extractor from being distorted, as evidenced by the performance drop observed in the first row. When cleansing is omitted, a significant decrease in performance is also observed.

Training time. To assess the efficiency of the proposed algorithm, we examine the correlation between training time and test accuracy as depicted in Figure 3d. We run FFT on the CIFAR-100 benchmark under 90% noisy ratio. We run each algorithm for 5 epochs. As described in Figure 3d, TURN shows better performance (+2▷69% compared to UNICON) while spending the smallest training time (i.e., ⇥0▷12 compared to UNICON). This is because leveraging noisy labels by giving pseudo-labels for them, such as DivideMix and UNICON, requires a significant amount of time due to their computationally expensive nature. Furthermore, robust loss-based algorithms exhibit reduced computational costs compared to the pseudo-label based methods but they used entire training samples, while TURN utilize part of a clearer training dataset. This analysis emphasizes the efficiency of TURN.

6 RelatedWork

Learning with noisy labels. Noisy label problem has been explored extensively in recent researches [Li et al., 2020; Cheng et al., 2021; Kim et al., 2021; Xia et al., 2022; Karim et al., 2022; Cheng et al., 2023]. Existing methods mainly address this problem by (1) detecting corrupted instances and only using label information of clean examples [Li et al., 2020; Cheng et al., 2021; Kim et al., 2021; Xia et al., 2022] (2) designing loss functions [Zhang and Sabuncu, 2018; Wang et al., 2019; Zhou et al., 2021] or regularization terms [Liu et al., 2020; Ko et al., 2022; Cheng et al., 2023] with robust behaviors. Recently, the majority of the researches [Zheltonozhskii et al., 2022; Karim et al., 2022; Li et al., 2022; Tu et al., 2023; Huang et al., 2023] have focused on applying self-supervised approaches to construct robust feature extractors on label noise. The authors of [Zheltonozhskii et al., 2022; Ko et al., 2023] used pre-trained models to run semi-supervised approaches with the initial parameters from the SimCLR [Chen et al., 2020] and showed significant performances. However, these approaches may be over-complicated requiring hyperparameter tuning for different datasets, as well as significant computation resources.

Pre-trained visual models. Recently, several studies have demonstrated that PTMs, which are trained on the large image set (ImageNet), can learn universal visual representations that are useful for downstream computer vision tasks. This has eliminated the need to train a new model from scratch. With the advancement of computational power and development of deep models such as ViT[Dosovitskiy et al., 2021] and ConvNext [Liu et al., 2022], the capabilities of PTMs have greatly improved. Utilizing PTMs has been considered as an effective solution for multi-modal models such as CLIP [Radford et al., 2021] and Data2Vec [Baevski et al., 2022], which can effectively represent various types of domains. As researchers make pre-trained weights of PTMs available to the open-source community, there is growing interest in finding ways to effectively use these pre-trained weights.

7 Conclusion

This study introduces TURN, an algorithm designed to handle noisy labels using large pre-trained models. It focuses on robustly leveraging these models and minimizing noisy label effects. TURN uses linear probing and fine-tuning on a refined subset of the training dataset. Its enhanced performance is evident from experiments on CIFAR-100, Clothing 1M, and WebVision datasets, demonstrating both improved results and lower computational costs.

Acknowledgments

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST),

10%) and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration, 90%)