Screening, Rectifying, and Re-Screening: A Unified Framework for Tuning Vision-Language Models with Noisy Labels

1 Introduction

In the field of image classification, label quality often depends on various factors, such as the complexity of distinguishing target objects and the expertise of annotators. In annotation systems like crowdsourcing, noisy labels are inevitable. Considering deep neural networks (DNNs) are highly susceptible to memorizing mislabeled training samples, addressing learning with noisy labels is essential for ensuring the robustness of learned DNNs. Recently, pre-trained vision-language models [Radford et al., 2021] have achieved remarkable progress and are increasingly adopted for downstream image classification tasks. This paper specifically tackles the challenge of fine-tuning pre-trained vision-language models under the presence of noisy labels.

Learning DNNs with noisy labels presents two main challenges. First, it requires effectively identifying mislabeled samples to prevent DNNs from memorizing incorrect data. Research [Arpit et al., 2017] reveals that DNNs tend to first memorize clean samples with dominant patterns before noisy samples with less representative patterns. Hence, the small loss criterion is widely used for separating clean samples from noisy samples. Methods such as co-training with two models [Li et al., 2020; Lu et al., 2021; Kim et al., 2024] leverage this criterion to mutually identify noisy samples. The second challenge is extracting value from noisy samples. Existing methods typically assign pseudo-labels to noisy samples [Li et al., 2020; Yao et al., 2021] or design self-supervised constraints [Liu et al., 2020].

∗Corresponding author (De Cheng)

Recently, several studies [Wu et al., 2023; Guo and Gu, 2024; Feng et al., 2024] have explored the challenge of tuning the vision-language models under noisy labels. For example, [Wu et al., 2023] analyzes the impact of noisy labels on CLIP [Radford et al., 2021], revealing that performance significantly deteriorates as noise increases. To address this, existing methods often rely on self-generated predictions to clean samples or assign pseudo labels, as seen in [Guo and Gu, 2024] and [Feng et al., 2024]. However, these approaches suffer from self-confirmation bias, where prediction errors propagate and amplify during training. Besides, conventional small-loss criterion fails to reliably distinguish clean samples from noisy ones, especially in ambiguous cases.

To address these limitations, we propose a novel algorithm that consists of three main steps: screening observed labels, rectifying potential noisy labels, and re-screening rectified labels. First, we introduce a dual-level semantic matching criterion that combines macro-level and micro-level textual prompts to better differentiate clean samples from noisy ones in the training loss space. This criterion is applied for identifying whether a training sample is a clean, ambiguous, or noisy sample. Second, we design separate label rectification strategies for refining labels of noisy and ambiguous samples with pseudo labels, aiming to effectively incorporate them into training. Lastly, we mitigate self-confirmation bias by re-screening rectified labels using cross-validation with another vision-language model, BLIP [Li et al., 2022]. Extensive experiments conducted on ten datasets, demonstrate that the proposed method achieves state-of-the-art performances in tuning vision-language models with noisy labels.

Key contributions of this paper are summarized as below:

1. We propose a novel unified framework, comprising three steps: Screening to identify noisy and ambiguous samples, Rectifying to assign corrected labels, and ReScreening to validate the rectified labels for improving model robustness.
2. A novel dual-level semantic matching method is introduced, using macro- and micro-level textual prompts to effectively distinguish noisy samples from clean ones.
3. Extensive experiments on ten datasets demonstrate that our method establishes a new state-of-the-art for tuning vision-language models with noisy labels.

2 RelatedWork

Learning with Noisy Labels. Noisy labels degrade robustness of DNNs due to their susceptibility to overfitting. To mitigate this, researchers have developed methods for noise detection and label cleaning. Leveraging the observation that DNNs learn clean samples faster than noisy ones [Arpit et al., 2017], several methods (e.g., [Park et al., 2023; Patel and Sastry, 2023; Kim et al., 2024]) utilize the smallloss criterion to identify clean samples. While effective to some extent, these methods can misclassify noisy samples resembling clean ones. Approaches like [Guo and Gu, 2024; Feng et al., 2024] employ label correction techniques after sample partitioning, but they are prone to self-confirmation bias due to their reliance on model predictions, potentially hindering performance.

The other type of methods uses training objectives based on label transfer matrices [Lin et al., 2024; Nguyen et al., 2024; Bae et al., 2024], reweighted losses [Yao et al., 2024], and self-supervised learning [Liu et al., 2020; Zhu et al., 2024]. However, these methods often exhibit limited effectiveness due to their reliance on specific noise patterns.

Training strategies like curriculum learning (e.g., [Yu et al., 2024]) and progressive label adjustment (e.g., [Zhang et al., 2021], [Chen et al., 2024]) mitigate noisy labels by gradually refining the learning process. While effective, these methods often entail increased computational overhead and necessitate careful hyperparameter tuning (e.g., epochs, curriculum structure). Improper optimization can lead to longer training times, higher resource consumption, and suboptimal performance.

Unlike existing methods, our method presents a threestep framework-screening, rectifying, and re-screening labels. We use a dual-level semantic matching criterion for improved label screening and distinct label rectification strategies for noisy and ambiguous samples. To mitigate selfconfirmation bias, we employ a pre-trained vision-language model to cross-validate rectified labels, enhancing label correction accuracy and robustness compared to methods relying on a single model for both detection and rectification.

Prompt Tuning for Vision-Language Models. Visionlanguage pre-trained models have advanced significantly in recent years. CLIP [Radford et al., 2021] employs prompt engineering to incorporate category-specific information into text inputs, allowing its pre-trained model to adapt to various tasks without further training. However, manual prompt design is time-consuming and requires expertise. To address this, CoOp [Zhou et al., 2022b] introduces learnable prompts optimized for specific datasets, reducing manual effort and improving task adaptability. CoCoOp [Zhou et al., 2022a] further enhances this by integrating image information into prompts using a lightweight network, enabling better context learning and generalization to unseen categories. BLIP [Li et al., 2022] leverages weak supervision to generate highquality visual information for task-specific purposes, improving performance. More recent works like ArGue [Tian et al., 2024] align prompts with visual attributes from large language models and apply negative prompting for better outof-distribution generalization, while AdvPT [Zhang et al., 2024] uses learnable text prompts aligned with adversarial image embeddings to enhance robustness. While these methods advance prompt tuning, they primarily focus on static or model-based prompt designs. In contrast, our approach aims to adapt the pre-trained model to noisy label tasks. We combine macro- and micro-level textual prompts for label screening and rectification, and use BLIP to cross-validate rectified labels. This approach improves model robustness and adaptability, complementing traditional prompt tuning techniques.

3 Preliminary

Problem Definition. This paper aims to address the adaptation of the vision-language pre-trained model using training data with noisy labels. Formally, we denote the training dataset as {(xi, yi)}Ni=1 , where xi denotes the i-th training sample, yi denotes the observed label of xi, and N represents the total number of training images. Supposing the number of classes be C, we have yi ∈ {1, · · · , C} which may be incorrect. The target is to adapt a pre-trained vision-language model which can tackle the image recognition task through matching images with class descriptions to the above dataset. Prompt Tuning for Vision-Language Model. Following [Zhou et al., 2022b], we use the vision-language pretrained model CLIP consisting of an image encoder and a text encoder, as the image classification backbone model. We apply the prompt tuning algorithm to adapt the pre-trained text encoder to the target dataset. Given an input image xi, the image encoder transforms it into a feature vector fi. For purpose of tuning the text encoder, a set of learnable prompts are used to provide extra context of each class's textual prompt. We denote the learnable prompts of the j-th class as tj. Feeding the j-th class's text descriptions and the learnable prompts tj into the text encoder, we can obtain the j-th class's embedding vector oj. Based on the image feature vector fi and all classes' embedding vectors, we can infer the probability of xi belonging to the j-th class as follows:

where τ is a hyperparameter set to 2. Through optimizing the learnable prompts only while freezing the image and text encoder, we can achieve the efficient adaptation of the pretrained model into the target dataset. This can take advantage of the prior knowledge of the pre-trained model in extracting generalized vision and textual features. Due to few parameters required for optimization, only a small number of training data is required during the model adaptation process.

4 Method

4.1 Overview of Proposed Framework

As shown in Fig. 1, we propose a framework leveraging pretrained vision-language models to tackle noisy label learning. It consists of three steps: 1) screening observed labels to identify potential mislabels, 2) rectifying these labels for effective optimization, and 3) re-screening to mitigate selfconfirmation bias.

In the screening step, we use a dual-level semantic matching mechanism with macro-level and micro-level prompts to classify samples as clean, ambiguous, or noisy ones. For rectification, noisy samples are assigned pseudo-labels, while ambiguous ones receive a combination of observed and pseudo labels. Finally, we use BLIP to re-screen and validate rectified labels. High-quality samples are directly used for prompt learning, while the rest are augmented via data mixing to minimize the influence of error labels.

4.2 Label Screening Based on Dual-level Semantic Matching

Dual-level Semantic Matching. During training, DNNbased models tend to memorize clean samples first, followed by noisy samples, which is why the small-loss criterion is commonly used to differentiate between the two. However, for CLIP prompt learning, using a single textual prompt struggles to effectively separate clean and noisy samples in the loss space. Macro-level prompts, based on class names, ensure inter-class separability but are prone to overfitting noisy samples due to the simplicity of the image-language matching. On the other hand, micro-level prompts, which describe fine-grained attributes, provide more reliable matching, reducing overfitting to noisy samples but may cause underfitting on clean samples. As a result, relying on just one type of prompt for image-class matching leads to significant overlap between clean and noisy samples. To address this, we propose a dual-level semantic matching mechanism, combining both macro-level and micro-level prompts as a label screening criterion. This approach facilitates the identification of clean, ambiguous, and noisy samples.

1. Macro-level Textual Prompt. We follow CLIP's original design, using a class name template to generate macro-level textual prompts. For class j, the macro-level textual prompt T mac j is defined as:

By feeding T mac j and learnable prompts tmac j into the text encoder, we obtain the embedding vector omac j for class j. According to Eq. 1, these class embeddings {omac j }Cj=1 are then used to compute the probabilities pmac i ∈ [0, 1]C , representing the likelihood of sample xi belonging to each class.

2) Micro-level Textual Prompt. Unlike macro-level prompts, micro-level prompts introduce class-specific features to enhance the alignment between image and text embeddings. Following [Feng et al., 2024], we generate these prompts by incorporating detailed features such as shapes, textures, colors, and other unique attributes of each class. For class j, the micro-level textual prompt T mic j is expressed as:

The text encoder processes T mic j and tmic j to produce the micro-level embedding omic j . Using these embeddings {omic j }Cj=1 , we can predict the class probabilities pmic i ∈ [0, 1]C , which represent the likelihood of xi belonging to each class.

3) Label Screening Criterion. Based on the class probabilities estimated from the macro-level and micro-level textual prompts, we define a label screening criterion with a samplewise loss function composed of three key terms: (1) a conventional cross-entropy loss, (2) a consistency constraint between the two types of prompt-induced predictions, and (3) an entropy penalty term. These terms are designed to enhance model robustness, especially in the presence of noisy labels.

The cross-entropy loss evaluates the discrepancy between the predicted class probabilities and the observed label. Specifically, for sample xi with observed label yi (one-hot encoded vector of yi), the loss is computed as:

where yi,j represents the j-th element of yi, and pmac i,j and pmic i,j are the j-th elements of pmac i and pmic i , respectively.

To address the issue of noisy labels, we impose a consistency constraint between the predictions generated by macrolevel and micro-level prompts. This is achieved by calculating the Jensen-Shannon divergence between the two probability distributions pmac i and pmic i . The consistency loss for sample xi is given by:

This term helps mitigate the impact of noisy labels by promoting agreement between the predictions from both levels, effectively reducing the overfitting to noisy samples. By aligning the macro-level and micro-level predictions, the model is less likely to overfit to mislabeled data, resulting in improved generalization.

The entropy penalty term encourages more tight predictions by penalizing overly smooth outputs. It is designed to reduce the overlap between clean and noisy samples. The entropy penalty for sample xi is calculated as:

The overall loss function is a weighted sum of the three loss terms:

where λ and β are hyperparameters.

Fig. 2 illustrates the loss distributions of clean and noisy samples using macro-level, micro-level, and dual-level textual prompts on the DTD dataset with 75% Pairflip label noise. The loss distributions of clean and noisy samples overlap significantly when using either macro-level or micro-level prompts alone. In contrast, the dual-level textual prompts effectively reduce the overlap between clean and noisy samples, highlighting the advantage of combining macro-level and micro-level prompts for more accurate label screening. Tri-Segment Sample Screening. Previous approaches [Guo and Gu, 2024; Arazo et al., 2019] typically use small-loss mechanisms to separate clean and noisy samples. However, noise complexity often leads to misclassification of clean samples with losses resembling noisy ones, or vice versa. To address this, we propose a tri-segment screening strategy that categorizes samples into clean, ambiguous, noisy classes, accounting for overlap between clean and noisy data.

We model sample-wise losses with a two-component Gaussian Mixture Model (GMM), fitting the model every two epochs. The first and second Gaussian components represent the clean and noisy data loss distributions, respectively:

where fc(l) and fn(l) are the probability density functions of clean and noisy data, respectively. µk and σk denote the mean and variance of the k-th Gaussian component, respectively.

Given the overlap between these distributions as illustrated in Fig. 3, we define the ambiguous region for samples with loss values in this range. For a confidence level θ, we determine the ambiguous region boundaries α1 and α2 as follows:

Solving these equations gives thresholds α1 and α2:

The lower and upper boundaries of the ambiguous region are ηl = min(α1, α2) and ηu = max(α1, α2), respectively. A sample xi is classified as follows:

• If ℓ(xi, yi) < ηl, xi is clean;
• If ηl ≤ ℓ(xi, yi) < ηu, xi is ambiguous;
• If ℓ(xi, yi) ≥ ηu, xi is noisy.

This label screening strategy partitions the dataset into clean, ambiguous, and noisy subsets, guiding further data utilization and label rectification for training.

4.3 Label Rectification and Re-screening

Label Rectification. To ensure sufficient training data, we introduce label rectification strategies that update each training sample's label using pseudo labels generated from model predictions. The pseudo label of sample xi, denoted as ˆyi, is the average of predicted probabilities from macro-level and micro-level prompts, i.e., ˆyi = (pmac i +pmic i )/2.

The updated label ˜yi is determined by the following rules:

• If xi is clean, ˜yi = yi.
• If xi is ambiguous, we combine the observed label yi and pseudo label ˆyi with confidence weight wi:

wi is determined by the posterior probability that xi belongs to the clean set.

• If xi is noisy, the observed label is likely incorrect, so the pseudo label ˆyi is used: ˜yi = ˆyi.

Label Re-screening. Using pseudo labels for rectification can introduce self-confirmation bias due to the lack of reliable quality evaluation. To mitigate this, we use the pretrained BLIP model for cross-validation. For each sample xi, we generate a text prompt ˜ Ti by incorporating the class with the highest value in ˜yi into the micro-level prompt template. We then use BLIP to compute the similarity between xi and ˜ Ti, denoted as si. The scores of all samples are fitted with a two-component GMM, and the posterior probability of si belonging to the component with the larger mean is calculated, resulting in a quality score qi. This score is used to separate the training samples into high-quality and low-quality sets:

where κ is a threshold, typically set to 0.5.

4.4 Training Objective

Considering the unreliability of samples in ˜Dl, we apply the mixup operation [Berthelot et al., 2019] to augment them, yielding an updated dataset ˜D′ l . This reduces the model's reliance on individual noisy labels and enhances data diversity, preventing overfitting. The final training objective is computed by accumulating losses over the union of ˜Dh and ˜D′ l :

This objective is used to optimize the learnable prompts, mitigating the influence of noisy samples while making full use of all training samples.

5 Experiments

5.1 Datasets

We evaluate our method on ten datasets, including Flowers102 [Nilsback and Zisserman, 2008], EuroSAT [Helber et al., 2019], StanfordCars [Krause et al., 2013], OxfordPets [Parkhi et al., 2012], DTD [Cimpoi et al., 2014], Caltech101

[Fei-Fei et al., 2004], UCF101 [Soomro, 2012], Food101 [Bossard et al., 2014], ImageNet [Deng et al., 2009] and SUN397 [Xiao et al., 2010]. Following JoAPR [Guo and Gu, 2024], we adopt the same noise-labeled ways to generate our noisy dataset. One is Symflip noise, which is generated by randomly drawing labels from other categories in the dataset to replace the true labels. The other is more challenging Pairflip noise, which is generated by exclusively selecting labels that are adjacent to the true label to replace it. Following CoOp [Zhou et al., 2022b], we sample a 16-shot training set from each dataset and employ the original test set for evaluation. To assess robustness, we test the method under noise rates ranging from 12.5% to 75%.

5.2 Implementation Details

Following the previous works [Zhou et al., 2022b; Guo and Gu, 2024], we adopt the CLIP model with ResNet-50 [He et al., 2016] as visual encoder and Transformer [Vaswani et al., 2017] as text encoder. The number of tokens in learnable prompts is set to 16. We adopt SGD optimizer to train our model with an initialization learning rate of 0.002 and apply cosine annealing strategy. The maximum number of training epochs is set to 200, except for ImageNet where it is set to 50. To mitigate gradient explosion in the early stages of training, we adopt a warm-up strategy and fix the learning rate to 1 × 10-5 in first training epoch. The hyper-parameters λ, β and θ, are set to 0.5, 0.001, and 0.01, respectively.

5.3 Comparison with State-of-the-art Methods

Table 1 presents a comprehensive comparison between our method and state-of-the-art (SoTA) approaches including CoOp [Zhou et al., 2022b], Robust UPL [Wu et al., 2023], and JoAPR/JoAPR* [Guo and Gu, 2024], across ten datasets under various noise levels. Our method achieves the highest accuracy in most scenarios and the highest average accuracy across all noise levels. It can be seen that CoOp demonstrates robustness at lower noise levels, but its performance deteriorates significantly as the noise ratio increases. Robust UPL [Wu et al., 2023] rely on pseudo-labels to alleviate the impact of noisy data on model performance. As the noise rate increases, the confidence in pseudo-labels decreases, which negatively impacts model performance. JoAPR improves upon CoOp by using data partitioning and label refurbishment but still struggles in high-noise scenarios. For instance, under 62.5% Symflip noise on the DTD dataset, our method outperforms JoAPR by 8.73%. On the StanfordCars dataset with 75% Pairflip noise, our method surpasses JoAPR* by 10.86%. Additionally, our method demonstrates strong robustness to high noise levels across all datasets.

5.4 Ablation Study

Effectiveness of Key Components. To better determine the contributions of key components in our method, we conducted a series of ablation experiments on the DTD and OxfordPets datasets, as shown in Table 2. The pre-trained vision-language model CoOp is used as the baseline (referred to as No.1). No.2, No.3, and No.4 represent the application of macro-level semantic prompt (MaLS), micro-level semantic prompt (MiLS), and dual-level semantic prompts (DLSM) to the baseline, respectively. Here, all training samples and their observed labels are directly used without processing. Comparing No.2, No.3, and No.4 highlights the effectiveness of using dual-level semantic matching for label screening. No.5 and No.6 demonstrate the importance of rectifying ambiguous and noisy labels selected out by our method, while No.7 and No.8 show that label re-screening further improves robustness. These contributions enhance our model's ability to handle various noise more effectively.

Comparison with Existing Two-Segment Label Screening and Rectification Methods. To evaluate the effectiveness of our proposed method, we compare it with two commonly used two-segment methods that categorize samples into clean and noisy categories: 1) keeping clean labels and refurbishing noisy labels as in Eq. (10) [Sohn et al., 2020; Feng et al., 2024], referred to as KCL-RNL; 2) refurbishing clean labels as in Eq. (10) and replacing noisy labels with pseudo labels [Li et al., 2020; Guo and Gu, 2024], referred to as RCL-RNL. As shown in Fig. 4, our method achieves superior performance across noise ratios on two datasets. This validates the advantages of tri-segment partitioning for precise label rectification and enhanced robustness through differentiated sample handling.

Confidence Weight Analysis. We compared our adaptive setting scheme of the confidence weight (wi) with fixed values (0.3, 0.5, 0.7). As shown in Table 3, the adaptive scheme consistently outperforms fixed weights by better handling uncertainty in ambiguous samples.

Robust Loss. Following [Wu et al., 2023], we evaluate CoOp with generalized cross-entropy (GCE) [Zhang and Sabuncu, 2018] on two noisy datasets. As shown in Fig. 4, while GCE improves CoOp's performance over cross-entropy, our method achieves better results in most cases.

Hyper-parameters Analysis. We analyze the impact of confidence threshold θ on model performance (Fig. 5). The model performs best at θ = 0.01, with performance degrading as θ increases due to misclassification of clean samples.

5.5 Extreme Noisy Analysis

To assess the robustness of our method, we conducted a challenging experiment where we introduced 100% noise into the Caltech101 and OxfordPets datasets, meaning all training data were deliberately mislabeled. As shown in Table 4, our method demonstrates exceptional performance in both Symflip and Pairflip noise scenarios, significantly outperforming JoAPR. These results highlight the effectiveness and robustness of our approach under extreme noise conditions.

6 Conclusion

We propose a framework for tuning vision-language models with noisy labels through screening, rectifying, and rescreening strategies. Our dual-level semantic matching mechanism partitions samples into clean, ambiguous, and noisy samples, while the label rectification steps assign pseudo labels to ambiguous and noisy samples, and the re-screening step reduces self-confirmation bias through cross-validation.

Extensive experiments show superior performance over prior methods. However, challenges remain under high noise rates and fine-grained settings, which we aim to address through noise-robust loss functions and curriculum learning.

Acknowledgements

This work is supported in part by the National Key R&D Program of China (No. 2024YFB3908503, 2024YFB3908500, 2022ZD0120100), in part by the National Natural Science Foundation of China (No. 62376206, 62176198, 62322608), in part by Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515010255), and in part by Fundamental Research Funds for the Central Universities (No. QTZX25083).