Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models

1 Introduction

Addressing downstream tasks with limited labeled data remains a critical challenge in real-world applications [Yang et al., 2023; Shao et al., 2026; Yang et al., 2025]. Two prominent paradigms have emerged to tackle this issue: (1) adapting pre-trained models, such as vision-language models (VLMs) [Radford et al., 2021; Jia et al., 2021], for downstream tasks; and (2) applying semi-supervised learning (SSL) methods that leverage large amounts of inexpensive unlabeled data. Recent work [Lv et al., 2025b] has further suggested combining them, where the strong zero-shot capabilities of pre-trained VLMs are used to generate reliable pseudo-labels that guide semi-supervised fine-tuning, thereby fully exploiting the strengths of both paradigms.

Recent efforts have explored integrating these two paradigms by leveraging pre-trained models to extract domain knowledge from unlabeled data for downstream task adaptation. A common approach is model ensembling (Figure 1.a), such as V-PET [Gupte et al., 2024], which aggregates multi-view one-hot predictions from multiple vision models. As shown in Figure 1.b, another line of work selects top-k predictions using class-wise or confidence-based thresholds. For instance, GRIP [Menghini et al., 2023] and UPL [Huang et al., 2022] enforce class balance by selecting equal samples per class; CPL [Zhang et al., 2024] refines labels via instance-level confidence matrices.

* Corresponding author.

However, these methods struggle to balance the trade-off between pseudo-label accuracy and bias. Ensemble-based approaches can produce highly accurate pseudo-labels, but they often inadvertently amplify biases inherited from pretraining, leading to uneven performance across subgroups-a phenomenon commonly referred to as the "Matthew effect" [Zhu et al., 2022; Chen et al., 2022; Gan and Wei, 2024; Guo et al., 2020]. In contrast, soft-labeling techniques and class-wise pseudo-label allocation strategies can alleviate overconfidence to some extent, but they tend to introduce additional label noise and are highly sensitive to manually tuned hyperparameters. To the best of our knowledge, adaptively generating reliable and unbiased pseudo-labels in response to model performance remains a critical yet underexplored challenge in semi-supervised fine-tuning.

To address this challenge, we propose a novel method named Bi-Consistency-Guided Self-Training (Bi-CoG). As illustrated in Figure 1.c, Bi-CoG leverages multiple VLMs as base learners to perform dynamic self-training guided by both inter- and intra-model consistency as well as model error estimation. Specifically, we conduct a dynamic self-training process that comprises three-stage pseudo-label selection. In the first stage, we apply majority voting to generate ensembled pseudo-labels, ensuring reliable supervision. Then, we assess each model's bias on a sample using weak and strong augmentations: predictions consistent under no and weak augmentation but changing under strong augmentation indicate reliability and less prone to pre-training bias. Finally, based on theoretical insights, we dynamically adjust the upper bound on the number of pseudo-labels used for training, according to models' relative accuracy on labeled data. This further filters pseudo-labels and enables adaptive self-training throughout the training process.

As a plug-and-play approach, Bi-CoG can be seamlessly integrated into existing VLM fine-tuning pipelines. To evaluate its effectiveness, we conduct experiments under both open-world generalization [Zhou et al., 2024] and standard semi-supervised learning[Yang et al., 2023] settings, combining Bi-CoG with various prompt-tuning methods. Extensive results demonstrate the effectiveness of our bi-consistencyguided dynamic self-training strategy, which consistently improves performance across 14 datasets without relying on larger models or external knowledge, outperforming state-ofthe-art (SOTA) methods.

Our main contributions are summarized as follows:

• We propose Bi-CoG, a plug-and-play method that jointly leverages the pre-trained knowledge of VLMs and external information from unlabeled data.
• We design a deeply coupled self-training strategy that integrates inter-model consistency, intra-model consistency and model error estimation, enabling reliable, lowbias pseudo-label generation and adaptive training.
• Theoretical analysis and extensive experiments demonstrate that Bi-CoG consistently improves performance, achieving up to a 5.69% gain across 14 datasets and surpassing SOTA methods.

2 RelatedWork

Parameter-efficient Fine-tuning on VLMs While pretrained VLMs demonstrate strong zero-shot generalization in domain-specific downstream tasks, their large parameter sizes often hinder efficient adaptation to specific applications [Zhou et al., 2022b]. To address this, recent works have proposed parameter-efficient fine-tuning approaches that introduce a small number of trainable parameters while keeping the majority of the model frozen [Li et al., 2024]. TipAdapter [Zhang et al., 2022] and CLIP-Adapter [Gao et al., 2024] append lightweight bottleneck layers after the text or image encoders to learn residual features for better representation. CoOp [Zhou et al., 2022b] introduces prompt tuning to VLMs for the first time by replacing manually crafted hard prompts (e.g., "a photo of a classname") with learnable soft text prompts optimized via backpropagation. Building on this idea, methods such as MaPLe [Khattak et al., 2023a], PromptSRC [Khattak et al., 2023b], and BMIP [Lv et al., 2025a] extend prompt tuning to both visual and textual encoders, enabling joint vision-language adaptation. While these methods have advanced the application of VLMs to downstream tasks, the transformation of prompts into continuous vectors makes them prone to overfitting and leads to sub-optimal performance in open-world scenarios [Zhou et al., 2022a; Khattak et al., 2023b]. Beyond prompt tuning, recent work on VLM selection and reuse [Tan et al., 2025] further explores how to choose and adapt pre-trained models for downstream tasks.

Pseudo-labeling in Semi-supervised Learning Semisupervised learning (SSL) addresses label scarcity by leveraging both labeled and unlabeled data. Pseudo-labeling, the most widely used SSL approach, assigns labels to unlabeled data based on model predictions and improves performance through self-training. The key to effective pseudolabeling lies in the generation of high-quality pseudo-labels [Yang et al., 2023]. Traditional SSL methods typically filter pseudo-labels using confidence thresholds or adopt data augmentation to generate diverse views, aiming to improve label quality and model robustness [Lee and others, 2013]. For VLMs, their strong generalization ability allows them to produce highly accurate pseudo-labels even in zero- or few-shot settings, making them strong candidates as pseudolabelers. Recent works have explored more advanced strategies: GRIP [Menghini et al., 2023] gradually increases the upper limit on the number of pseudo-labels to mitigate calibration errors [Oh et al., 2024] and label imbalance [Wang et al., 2022; Guo and Li, 2022]; and CPL [Zhang et al., 2024] progressively refines pseudo-labels based on a confidence score matrix to avoid overconfidence. Despite their effectiveness, these methods struggle with balancing pseudolabel noise and inherited pre-training biases [Oh et al., 2024; Wang et al., 2022; Li et al., 2021], while failing to dynamically adapt to model performance in open environments [Guo et al., 2025], ultimately limiting their potential.

Problem Definition Based on the aforementioned challenges in pseudo-labeling-specifically the issues of label noise and static adaptation-we formally define our task. In downstream scenarios with scarce labeled data, we typically have access to a small labeled dataset denoted as DL = {(xi, yi)}Mi=1 , along with a large amount of unlabeled data DUL = {xi}Ni=1 , where M ≪ N. Given a collection of K pre-trained VLMs {fk}Kk=1 (with K ≥ 3), the objective is to effectively leverage both DL and DUL to fine-tune the models and maximize performance on the target dataset Dtest.

3 Methodology

To address the prevalent challenges of balancing pseudo-label accuracy with model bias and reducing the strong reliance on empirical parameter tuning in prior methods, we propose

Algorithm 1 Bi-CoG's overall learning algorithm.

Input: Pre-trained VLMs {fk}Kk=1, labeled data DL, unlabeled data DUL, scaling factor α Output: Fine-tuned VLMs 1: for j = 1 toK do 2: e′j ← 0.5, updatej ←False 3: end for 4: t ← 0, improve ←True 5: while improve do 6: t ← t + 1 7: for j = 1 to K do 8: updatej ← False 9: ej ← MeasureError(DL, {fk}Kk̸=j ) 10: if ej < e′j then 11: Dj Inter ← InterConsistency(DUL, {fk}Kk̸=j ) 12: Dj Intra ← IntraConsistency(DUL, {fk}Kk̸=j ) 13: updatej ←ErrorAware (ej, e′j, t) 14: DjPL ← Subsample(Dj Inter ∩ Dj Intra , ( e′j ej )αtL′j) 15: end if 16: if updatej then 17: e′j ← ej, fj ←FineTune(fj,DL ∪DjPL) 18: end if 19: end for 20: if update == [False] ∗ K then 21: improve ← False 22: end if 23: end while 24: return {fk}Kk=1

include the pair (xi, y∗ ) in Dj Inter . Otherwise, the sample is discarded. Formally, this candidate set is defined as:

where Y denotes the label space, and I[·] is the indicator function. This strategy ensembles multiple models to generate reliable pseudo-labels, reducing noise from individual prediction errors. Moreover, the leave-one-out scheme prevents amplifying biases inherent to the target model, thereby resulting in more accurate and unbiased data for fine-tuning.

Bi-Consistency-Guided Self-Training (Bi-CoG), a simple yet effective plug-and-play framework. As illustrated in Figure 2, the Bi-CoG framework comprises the following three major components for pseudo-label selection:

• Amodel ensembling module that aggregates predictions through majority voting based on inter-model consistency, generating high-quality candidate pseudo-labels to ensure reliable supervision.
• A multi-view filtering mechanism that leverages intramodel consistency by applying diverse transformations to unlabeled data and discarding biased samples with inconsistent predictions, thereby mitigating the influence of pre-training bias.
• An error-aware bounding strategy that dynamically estimates model performance and sets an upper limit on the number of pseudo-labels used, promoting stable and progressive self-training.

The procedure of Bi-CoG is summarized in Algorithm 1. We elaborate on each component in the subsequent sections.

3.1 Inter-model Consistency

During self-training with pseudo-labels, label fidelity is critical to effective and stable optimization [Angluin and Laird, 1988]. Excessive noise can mislead learning, causing convergence to suboptimal representations. To mitigate this issue, we first generate pseudo-labels based on inter-model consistency by integrating predictions of multiple VLMs, with the primary goal of ensuring the accuracy of pseudo-labels.

Concretely, for each model fj (1 ≤ j ≤ K), we construct a candidate pseudo-labeled dataset Dj Inter by selecting unlabeled instances on which the majority of the other K - 1 models reach a consensus. For each unlabeled sample xi ∈ DUL, we first compute the class-wise vote count from the other models as follows:

If the most voted label y∗ = argmaxy∈Y vi(y) satisfies vi(y∗ ) ≥ ⌈ K-1 2 ⌉, we assign xi a pseudo-label ˆ yi = y∗ and

3.2 Intra-model Consistency

While inter-model consistency improves pseudo-label reliability by aggregating agreement across models, it may still preserve instances where VLMs share similar biases, often due to overlapping pre-training corpora or architectural homogeneity [Gan and Wei, 2024; Wang and Russakovsky, 2023]. To further mitigate biased supervision, we introduce intra-model consistency, which leverages the consistency of a single model's predictions under different input views to refine the pseudo-labeled sets obtained in the previous stage.

Specifically, for each unlabeled sample xi ∈ DUL, we apply both weak and strong data augmentations. Each model fk then produces predictions on the original input xi, the weakly augmented ˜xwi , and the strongly augmented ˜xsi . A sample is retained only if the prediction remains consistent between the original and weakly augmented inputs, i.e., fk(xi) = fk(˜xwi ), but differs under strong augmentation, i.e., fk(˜xsi ) ̸= fk(xi). This criterion identifies samples with low uncertainty and meaningful sensitivity, as evidenced by prediction consistency under weak perturbations and inconsistency under strong perturbations. The intra-model consistent pseudo-label set for fj is defined as the intersection of predictions from the other models:

Once the Dj Intra are computed, the candidate pseudo-label set is constructed by intersecting its inter-model consistent set Dj Inter with the intra-model consistent set:

The intersection of pseudo-labels from inter-model consistency and intra-model consistency preserves accuracy while mitigating shared model biases. Consequently, the resulting pseudo-labeled datasets are both highly reliable and low in bias, providing a robust foundation for the subsequent dynamic self-training stage.

3.3 Error-aware Filtering

After bi-consistency-guided pseudo-labeling, we obtain a candidate sample set DPL, which contains high-accuracy and low-biased pseudo-labels. However, during self-training, it is crucial to determine an appropriate upper bound on the number of pseudo-labeled samples based on model performance. Prior studies [Angluin and Laird, 1988; Chen et al., 2022] have shown that even minor label noise, if not aligned with model competence, can significantly degrade performance, making further noise reduction in DPL essential. Existing methods [Menghini et al., 2023; Huang et al., 2022] typically rely on fixed thresholds to limit the number of pseudo-labels, which can be suboptimal as model performance evolves over time and across different datasets.

To address this limitation, we dynamically adjust the upper bound on pseudo-label usage based on estimated model error ratio, guided by a theoretical analysis. Specifically, following prior work [Angluin and Laird, 1988; Zhou and Li, 2005] we formalize the influence of noisy supervision on model fj during iterative self-training:

Lemma 1. Let Lt denote the training set Dtrain for model fj at iteration t, and et represent the relative error rate of the remaining K - 1 models at the same iteration, i.e., the probability that the majority vote produces an incorrect label. Then, training fj on Dtrain will result in performance improvement if the following condition is satisfied:

According to Lemma 1, the model will be improved throughout the training process as long as Equation 5 holds. In practice, we estimate the theoretical error rate ratio from the performance of the K-1 models on the labeled set. However, this estimation can be biased, as repeated training on labeled data tends to produce stable-and often low-error ratios, which are reflected in Equation 5 as values close to 1. In contrast, the actual error ratio on unlabeled data typically decreases during training, making the true ratio et et-1 lower than the estimated ˆ et ˆ et-1 . Ignoring this discrepancy would introduce bias into the estimated pseudo-label budget, compromising the validity of the theoretical analysis. To address this issue, we introduce Theorem 1 to provide a correction:

Theorem 1. As the training progresses, the true error rate ratio can be approximately equal to the t-power of the estimated error rate ratio, as shown below:

where α > 0 is a scaling factor that reflects the error reduction rate. Then, the original condition in Equation 5 is approximately equivalent to the following constraints:

Based on the theoretical analysis, we establish the necessary condition under which the model can be effectively optimized by self-training, as described in Equation 7, and establish an upper bound on the pseudo-label budget for each iteration. Here, L′j denotes the size of the pseudo-labeled training set used by model fj in the previous iteration. Accordingly, the final set of pseudo-labeled samples is formalized as:

This dynamic filtering strategy reduces the mismatch between model capacity and pseudo-label quantity. Consequently, the model can self-regulate the amount of pseudolabeled data during training, leading to stable and consistent performance improvements.

After training, Bi-CoG adopts a majority voting strategy to generate predicted labels for evaluation during the testing phase, ensuring robust and reliable assessment results.

In summary, Bi-CoG offers the following key advantages:

1. Plug-and-Play Adaptability: Bi-CoG can be seamlessly integrated with existing pre-trained models, providing consistent performance improvements with minimal changes to the original method.
2. Robust Pseudo-Labeling: By jointly leveraging biconsistency and error-aware mechanisms, Bi-CoG generates more reliable pseudo-labels, effectively reducing label noise and alleviating model bias.
3. Adaptive Self-Training Control: Empowered by the error-aware mechanism, Bi-CoG can automatically regulate the self-training process, eliminating the need for manually preset iteration counts and enhances adaptability across different scenarios.

4 Experiments

In this section, we design experiments to answer the following research questions:

1. RQ1: Does Bi-CoG enhance existing PEFT methods for VLMs in open-world generalization scenarios?
2. RQ2: Can Bi-CoG deliver competitive performance compared to conventional SSL approaches?
3. RQ3: How does Bi-CoG compare to other methods that jointly leverage pre-trained models and unlabeled data?
4. RQ4: What is the contribution of each component within the Bi-CoG framework?

4.1 Experimental Settings

Evaluation Paradigm. To answer RQ1 and RQ2, we evaluate the effectiveness of Bi-CoG on two widely adopted settings: open-world generalization setting [Zhou et al., 2024;

Lv et al., 2025a] and semi-supervised setting [Lee and others, 2013; Yang et al., 2023]. In open-world generalization setting, the model is trained only on a subset of categories (referred to as base classes), while the remaining categories are treated as novel classes. We report the harmonic mean (HM) of the model's performance on base and novel classes, as well as the overall accuracy on the full test set, which includes both base and novel classes. Semi-supervised setting follows a common protocol where only a small portion of the data is labeled, while the majority remains unlabeled.

Dataset and Baselines. We evaluate the performance of our method on 14 recognition datasets, including 11 commonly used for assessing VLMs and 3 classical benchmarks for semi-supervised learning. We select CLIP [Radford et al., 2021] and its three representative prompt tuning methods as baselines: CoOp [Zhou et al., 2022b], MaPLe [Khattak et al., 2023a], and PromptSRC [Khattak et al., 2023b], among which PromptSRC represents the current SOTA method. For methods that integrate the two paradigms, we select GRIP [Menghini et al., 2023] and CPL [Zhang et al., 2024] as baselines, with CPL representing the current SOTA approach.

Implementation Details. We set the number of VLMs K = 3 throughout all experiments. The setting K ≥ 3 ensures that an absolute majority can be achieved in most cases, facilitating reliable pseudo-label selection. While a larger K further improves stability, it also increases GPUmemory consumption. We adopt K = 3 to balance effectiveness and efficiency. For the scaling factor α, it provides a controllable mechanism to adjust the strictness of pseudo-label filtering: a smaller α enhances the utilization of more pseudo-labels, while a larger α suppresses noisy pseudo-labels by raising the filtering threshold. We uniformly set α = 1 across all datasets to ensure generalizability without dataset-specific tuning. All experiments are conducted on a single NVIDIA A800 GPU.

4.2 Experimental Results

RQ1: Does Bi-CoG enhance existing PEFT methods for VLMs in open-world generalization scenarios?

To evaluate the robustness of Bi-CoG in open-world scenarios, we conduct a comprehensive study by integrating it with three representative prompt tuning methods across 14 datasets. Notably, in all experiments related to Bi-CoG, we controlled the random seeds during both the initialization and training phases for different models to ensure diversity in the initial training stages. For each dataset, we report two metrics: HM of base and novel class accuracies, and overall accuracy (Acc.). We set the base-to-novel class ratio to 50:50 and report the mean and standard deviation over three runs. As shown in Table 1, Bi-CoG consistently improves both metrics across all datasets, achieving the highest average performance. Notably, when applied to CoOp, Bi-CoG yields over a 5% improvement in HM. Moreover, Bi-CoG demonstrates particularly strong gains on traditional openworld benchmarks. For example, on low-resolution datasets such as CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2009], it achieves up to an 8% increase in HM, underscoring its effectiveness across datasets with varying resolutions. These results highlight the effectiveness of Bi-CoG and emphasize the importance of integrating unsupervised strategies with pre-trained models in open-world settings.

RQ2: Can Bi-CoG deliver competitive performance compared to conventional SSL approaches?

As shown in Table 2, SSL algorithms exhibit a significant advantage over baseline prompt tuning methods on SSL datasets. This is largely due to the lower resolution of these datasets, which makes it more difficult for VLMs to effectively capture visual content [Krizhevsky et al., 2009; Lv et al., 2025b]. Notably, this advantage also comes at the cost of substantial computational overhead: conventional SSL methods typically require over 10 hours of training for the CIFAR dataset, whereas Bi-CoG achieves comparable or superior performance with only 1 hour of training under the same hardware configuration. This limitation of high computational cost, coupled with the sensitivity to the number of shots (increasing the number of shots from 4 to 25 yields much larger performance gains for SSL algorithms compared to VLM-based prompt tuning), highlights the practical challenges of traditional SSL methods. In contrast, Bi-CoG provides the model with more reliable training examples via adaptive pseudo-labeling, enabling prompt tuning methods to achieve SOTA performance under the challenging 4-shot setting while maintaining remarkable computational efficiency.

These results demonstrate that Bi-CoG can effectively leverage unlabeled data to capture richer visual representations even in low-resolution and data-scarce scenarios, while also significantly reducing training time compared to SSL methods, making it more practical and scalable.

RQ3: How does Bi-CoG compare to other methods that jointly leverage pre-trained models and unlabeled data?

To further validate the effectiveness of Bi-CoG, we conducted comparative experiments against two SOTA methods: GRIP [Menghini et al., 2023] and CPL [Zhang et al., 2024]. Specifically, GRIP defines a fixed epoch-dependent expression for selecting the number of pseudo-labels, while CPL relies on manually set confidence thresholds to filter pseudo-labels iteratively. Following the experimental protocol of GRIP, we adopted the same base-to-novel class ratio (62:38) and used CoOp as the underlying prompt tuning framework for Bi-CoG and all compared methods. This ensures a fair comparison by eliminating discrepancies from different base models or tuning strategies. As shown in Table 3, Bi-CoG achieves the best performance under both SSL

and open-world settings, outperforming CPL by up to 5.9 percentage points. Notably, this superiority stems from BiCoG's adaptive pseudo-label selection strategy during iterations: unlike GRIP's epoch-dependent rule or CPL's manual threshold tuning, Bi-CoG adjusts pseudo-label selection based on model consistency and estimated error rate, ensuring high-quality supervision signals. More importantly, Bi-CoG dispenses with the requirement for manually tuning hyperparameters, effectively reducing dependence on human expertise. In summary, Bi-CoG represents one of the most effective integrations of pretrain-finetuning and semi-supervised learning methods to date.

4.3 Ablation Study

To answer RQ4, we first examine the impact of the biconsistency-guided pseudo-labeling strategy on both pseudolabel accuracy and the distribution of biased samples. As shown in Figure 3.a, removing the inter-model consistency module results in a notable decline in pseudo-label accuracy, highlighting the importance of ensemble-based labeling for generating high-quality supervision. Notably, although the bi-consistency strategy exhibits relatively lower pseudo-label accuracy in the early stages of training, its accuracy steadily improves as training progresses. In contrast, other single-strategy pseudo-labeling methods show either a decline or fluctuation in accuracy, demonstrating the stability and reliability of bi-consistency-guided pseudo-labeling. Furthermore, as illustrated in Figure 3.b, incorporating intramodel consistency helps smooth the class-wise distribution of pseudo-labels, effectively mitigating the bias inherited from pre-training. These results collectively confirm that the biconsistency-guided strategy produces pseudo-labels that are both more accurate and less biased-two critical factors for stable and effective self-training.

To further validate the theoretical foundation of our erroraware mechanism, we analyze the evolution of the actual error rate ratio on pseudo-labeled samples and its estimation from labeled data on StanfordCars [Krause et al., 2013] under the open-world generalization setting. As illustrated in Figure 4, the condition in Equation 6 holds reliably during training, supporting the practical validity of our theoretical derivation and confirming the soundness of Bi-CoG.

Finally, we conduct a comprehensive ablation study on key components of Bi-CoG, using CoOp as the base model and reporting the average performance across 14 datasets. As shown in Table 4, any component of our three-stage pseudolabeling strategy, which integrates bi-consistency and erroraware filtering, plays a critical role in improving model performance. This highlights the deeply coupled and effective design of the pseudo-labeling mechanism in Bi-CoG.

5 Conclusion

This paper investigates the integration of pre-trained VLMs with semi-supervised learning for downstream tasks with sparse labeled data. We identify critical issues in existing approaches, including the trade-off between label quality and model bias, as well as the limited adaptability arising from the reliance on manually set fixed thresholds. To address these challenges, we propose Bi-CoG, a plug-andplay self-training method that incorporates a deeply coupled pseudo-labeling strategy with error-aware dynamic filtering capabilities. Theoretical analysis and experimental results across 14 datasets in both open-world generalization and semi-supervised settings demonstrate that Bi-CoG can be seamlessly integrated with any pretrain-finetuning methodologies, consistently achieving performance improvements.

Acknowledgments

This research was supported by the Key Program of Jiangsu Science Foundation (BK20243012), the National Science Foundation of China (62306133), and the "111 Center" (No. B26023).