VCC-INFUSE: Towards Accurate and Efficient Selection of Unlabeled Examples in Semi-supervised Learning

1 Introduction

Deep neural networks underpin various machine learning applications, with their success attributed in part to extensive labeled datasets like ImageNet [Deng et al., 2009] and COCO [Lin et al., 2014]. However, the process of collecting and annotating large datasets is resource-intensive and raises privacy concerns, making the acquisition of unlabeled data a more feasible and cost-effective alternative.

To address the challenge of limited labeled examples, semi-supervised learning (SSL) has gained prominence for leveraging abundant unlabeled data. Pseudo-labeling is a common SSL approach, as demonstrated by FixMatch [Sohn et al., 2020]. FixMatch generates pseudo labels for unlabeled data based on model predictions. The threshold-based selection module in FixMatch filters examples with confidence scores surpassing a fixed threshold τ for training. Formally, the loss on unlabeled data is defined as:

where L(ˆ ci, ˜ci) represents the loss between class label and confidence distribution.

Despite FixMatch's wide adoption, it may encounter challenges in utilizing unlabeled examples effectively and efficiently. Specifically, (1) Incorrect pseudo labels may arise due to calibration errors in model predictions, leading to unreliable performance. (2) The significant computation cost involved in forwarding the entire unlabeled dataset for pseudo label selection may be alleviated by dynamically pruning the dataset. This ensures that only informative data points contribute to the model's decision boundary, reducing computation overhead and accelerating convergence.

In this paper, we propose solutions to address these challenges, enhancing the reliability and efficiency of SSL methods based on pseudo-labeling. To address the first issue, we introduce Variational Confidence Calibration (VCC), a method aimed at obtaining calibrated confidence scores for pseudo label selection. The calibrated confidence score, closely aligned with the ground-truth probability of correct predictions, serves as a more reliable metric for choosing pseudo-labeled examples. While confidence calibration is a well-explored concept in fully-supervised settings, its application in semi-supervised learning (SSL) is more challenging due to the absence of ground-truth labels. To overcome this challenge, we utilize three consistency scores to assess prediction stability. By simultaneously considering both stability and confidence, we approximate calibrated confidence scores. Additionally, a variational autoencoder enhances stability by reconstructing the calibrated confidences.

To address the second issue, we propose INFUSE (Influence Function-based Unlabeled Sample Elimination), a method leveraging influence functions [Koh and Liang, 2017] to compute the importance of each unlabeled example. INFUSE dynamically retains data points with the highest importance, forming a smaller core set to replace the entire dataset. This core set allows for faster model convergence, reducing computation costs during training. The combined VCC-INFUSE method enhances prediction accuracy while minimizing training costs.

In summary, this paper makes following contributions:

• We propose the VCC method, which generates wellcalibrated confidence scores for more accurate pseudo labels, enhancing model accuracy. As a flexible, plugand-play module, VCC can be seamlessly integrated with existing SSL methods.
• We introduce the INFUSE method, which dynamically prunes unimportant unlabeled examples to expedite convergence and reduce computation costs during training.
• The effectiveness of our methods is demonstrated across multiple datasets and various settings.

2 RelatedWork

Semi-supervised Learning. FixMatch [Sohn et al., 2020] stands out as one of the most widely adopted SSL methods. FixMatch utilizes a weakly-augmented unlabeled example to obtain a one-hot pseudo label, followed by training the model on strongly-augmented examples to produce predictions consistent with the pseudo label. FlexMatch [Zhang et al., 2021] introduces an adaptive threshold strategy, tailored to different learning stages and categories. SimMatch [Zheng et al., 2022] considers both semantic and instance similarity, promoting consistent predictions and similar similarity relationships for the same instance. Additionally, explicit consistency regularization is employed in various SSL methods [Laine and Aila, 2016; Berthelot et al., 2020; Miyato et al., 2019; Ganev and Aitchison, 2020; Chen et al., 2023; Li et al., 2021; Feng et al., 2024; Lee et al., 2021].

Confidence Calibration. [Guo et al., 2017] identified the calibration problem in modern classifiers and proposed Temperature Scaling (TS) to rescale confidence distributions, preventing over-confidence. Ensemble TS [Zhang et al., 2020] extends TS's representation ability by expanding the parameter space. Additionally, [Kumar et al., 2018] introduces the MMCE method, a trainable calibration regularization based on Reproducing Kernel Hilbert Space (RKHS). Notably, these methods are designed for fully-supervised settings where ground-truth labels are available.

Core Set Selection. While most core set selection methods focus on the fully-supervised setting, our work aligns more closely with the semi-supervised learning context. [Paul et al., 2021] proposes the EL2N method, measuring the importance of an example based on the norm of the loss. EL2N significantly reduces training time with a minimal impact on accuracy. GradMatch [Killamsetty et al., 2021a] extends the core dataset to a weighted set using a submodular function. In the realm of SSL, RETRIEVE [Killamsetty et al., 2021b] addresses core set selection as an optimization problem. However, RETRIEVE's optimization function only considers the loss labeled set, potentially deviating from the desired objective of minimizing loss on the validation set.

3 Confidence Calibration with VCC

Many existing calibration methods are ill-suited for SSL due to the absence of ground-truth labels for unlabeled examples. Directly using the original confidence score for pseudo label selection can yield noisy results. To tackle this challenge, we introduce three different consistency scores (sens , stem, and sview ) to simultaneously gauge the stability of predictions. By combining these three scores, we obtain the approximated calibrated confidence ˜ r, which is closer to the probability of an example being correctly classified. However, ˜ r is not directly utilized for pseudo label selection, as the process of estimating ˜ r from three consistency scores can still be unstable for some examples.

To mitigate this instability, we introduce a Variational Autoencoder (VAE) to reconstruct ˜ r for pseudo label selection. The graphical model and framework illustration of VCC are provided in Fig.1 and 2, respectively. The VAE is learned jointly with the original classifier during training, where ˜ r serves as the "ground-truth" for calculating the reconstruction loss. For pseudo label selection, we leverage the output of the VAE as the calibrated confidence.

3.1 Ensemble Consistency

From a Bayesian perspective, the parameters θ of a model are sampled from a probability distribution over the training set D. Model's prediction for sample x can be formulated as:

where p(y|x, θ) represents the probability distribution of the label y of x given the parameters θ, and p(θ|D) represents the probability distribution of the model parameters θ trained on the dataset D. A single model may provide incorrect predictions, for example, x due to randomness and noise, even if the confidence is high. Considering the entire parameter space, if all model parameters yield consistent predictions for x, the result is more convincing. In this case, the prediction can be viewed as an ensemble of predictions from multiple models.

However, due to the large parameter space of θ, direct computation of Equation 2 is intractable. Therefore, we apply Monte-Carlo Dropout [Gal and Ghahramani, 2016] on the linear layer to approximate the computation of Equation 2. The feature map is cloned by K copies, followed by a Dropout layer to randomly eliminate neural connections in the classification head to obtain predictions. By doing so, the model would generate K estimated confidence distributions of example i, and the expectation can be treated as the ensemble of K different models:

Then, entropy is employed as the ensemble-consistency score to measure the different models' consistency of example: sens = - ∑Mc=1 ˜yc log ˜yc.

3.2 Temporal Consistency

In SSL, parameters are updated frequently during training, making the decision boundaries change all the time. Some examples may shift from one side of the decision boundary to the other side after parameter updates, bringing a change in classification results. In this case, the prediction results of many examples may be rather unstable. If these examples are used in training, it may result in incorrect pseudo labels and hinder the model's performance.

To measure the stability of prediction results between different steps, we propose the temporal consistency score, which considers the changes in the confidence distribution of an example between different epochs. Specifically, let yt represent the confidence distribution of an example at epoch t. The temporal consistency score can be calculated as:

where DKL represents the Kullback-Leibler Divergence, M is the number of classes, K represents the window size. In experiments, we empirically set K = 1 to preserve the sensitivity of abnormal confidences. Although both consider the problem from the perspective of time, our method differs a lot from the method proposed by [Zhou et al., 2020].

3.3 ViewConsistency

Multi-view learning [Xu et al., 2015] aims to leverage multiple perspectives to predict data, allowing different predictors to correct predictions collectively.

In semi-supervised learning (SSL), obtaining models with different views often involves dividing the entire dataset into multiple subsets for training multiple models. However, this incurs high model training costs, and the volume of labeled data in each subset may be too small to train a decent model. To address this, we use Exponential Moving Average (EMA) to construct models with different views. The original model parameter θ is updated using gradient descent, while θema is updated using the EMA scheme:

where β is a decay hyperparameter. These can be treated as two different views from the same network structure.

A typical classification model consists of a feature extraction network (backbone) and a classification head (linear layer). To increase the difference between two views, we adopt a cross-feature trick. The backbone of each view first extracts features from input, which are then fed into the classification head of another view. This can be formulated as:

After obtaining the outputs, the Kullback-Leibler (KL) divergence is used to measure the consistency between them:

It may seem like temporal consistency and view consistency overlap to some extent, as the predictions of the EMA model used in view consistency can also be considered an ensemble of predictions from past epochs. The difference is that the cross-feature trick is used in the computation of view consistency, which enforces this metric to focus more on consistency over multiple views rather than multiple time steps.

3.4 Approximation of Calibrated Confidence

We have introduced three scores to evaluate the stability of predictions. However, sens , stem, and sview cannot be directly used for pseudo label selection, which is based on confidence scores. To address this, we propose a simple method to approximate calibrated confidence with the three consistency scores.

The consistency scores are first normalized and summed up as the stability score. First, a fixed-length queue q is maintained to record the historical predictions of the unlabeled samples in mini-batches. Since sens , stem, and sview have different distributions, we normalize them with max-min normalization. Let u be the unlabeled example, the normalization is done as follows:

where t = {ens, tem, view}. After normalization, ˜ sens u , ˜ stem u , and ˜ sview u are all real numbers ranging from 0 to 1. These consistency scores evaluate the stability of the examples. However, some hard-to-learn examples may also have stable predictions but low confidence scores. Hence, the three consistency scores are not enough to describe the reliability of the prediction. To address this, the original confidence score of the sample ˜ sconf u = max (yu) is also used. Thus, an unlabeled sample u can be represented by a quadruple (˜ sens u , ˜ stem u , ˜ sview u , ˜ sconf u ).

The next problem is how to combine these four scores together for estimation. To avoid complex parameter tuning, VCC adopts a simple yet effective approach: taking the sum of their squares:

According to the results in [Guo et al., 2017], calibration errors mainly occur in the middle range of confidences, while samples with extremely low or high confidences tend to have smaller calibration errors. Therefore, we approximately treat the lowest/highest confidence score in q as well-calibrated and employ interpolation to calculate the calibrated confidence scores for other examples. To further eliminate the unfairness between different categories, the interpolation operation only considers examples with the same pseudo labels as the current example u.

and ˜ ru can be formulated as:

3.5 Reconstruct ˜ ru with Variational Autoencoder

In Section 3.4, we combined three consistency scores to obtain ˜ ru, which is the approximation of calibrated confidence scores. However, it may face instability due to the update of queue q and abnormal interpolation endpoints. To address this, we reconstruct the statistical-based ˜ ru in a learningbased way. Specifically, a Variational Autoencoder (VAE) is employed to generate the calibrated confidence score ru for pseudo label selection, and ˜ ru is used as input for training the VAE.

We assume r is generated by the following random process, which includes two steps: (1) a hidden variable z sampled from a prior distribution pθ(z); (2) a value r generated from the conditional distribution pθ(r|c, z, x):

However, the marginal likelihood pθ(r|c, x) is generally intractable. Hence another distribution qϕ(z|c, x) is introduced as the approximation of pθ(z) (Please refer to Appendix 1 A for details):

The first term is the likelihood of calibration reconstruction (denoted as Lrecon VCC ), where qϕ(z|c, x) is the encoder to infer the hidden variable z, and pθ(r|c, z, x) is the decoder to recover a calibrated confidence r. To compute the reconstruction loss, the approximated ˜ r is used as the ground truth. Besides, z needs to be sampled from qϕ(z|c, x). Reparameterization trick [Kingma and Welling, 2014] is used to predict the mean and standard deviation of z. By setting ϵ ∼ N(0, 1), the reparameterization is formulated as z = µ(c, x) + ϵ · σ(c, x). For the second term, under the Gaussian assumptions of the prior pθ(z|c, x) ∼ N(0, 1) and the approximator qϕ(z|c, x) ∼ N (µ(c, x), σ2 (c, x)), we have:

The overall objective function can be formulated as:

Although a more accurate confidence score is generated by combining three consistencies, it is still not as optimal as the inaccessible ground-truth. This is because there are many other "nuisance" and untraceable factors that affect the pseudo label's approach toward the ground-truth, such as the randomness of the neural networks. Under these circumstances, directly approaching the unreliable target may still degrade performance. The original VAE is proposed to learn continuous distribution features from discontinuous distributions by sampling a hidden variable. This process is suitable for suboptimal pseudo label learning because the approach of the prediction to the generated pseudo label can be viewed as the process of the prediction approaching the ground-truth. Since eliminating those nuisance factors cannot be tractable, we use VAE to simulate this process instead of the MLP.

1 Appendix is available at https://arxiv.org/abs/2404.11947.

4 Core Set Selection with INFUSE

In the previous section, we introduced VCC framework, which ensures well-calibrated confidence scores to improve accuracy in pseudo label selection. Nonetheless, as discussed earlier, training the SSL model still encounters substantial computational expenses. Furthermore, the incorporation of additional encoder and decoder of VCC introduces an extra computation overhead. To address these challenges, we present INFUSE-a core set selection methodology aimed at efficient example selection. Based on the influence function [Koh and Liang, 2017], INFUSE allows for training the SSL model using only a subset of the complete unlabeled dataset, so that training time can be significantly reduced.

In SSL, the model should minimize the loss on the validation set to obtain the highest generalization accuracy:

Here H is the loss function, τ is the threshold for pseudo label selection, q is the confidence distribution, ˆ q is the pseudo label, and R(θ) is the total loss on labeled dataset S and unlabeled dataset U. Now assume the weight of an unlabeled example u′ is increased by ϵ. Denote LU(u′ , θ) = λ · 1 (max(qu′ ) ≥ τ ) · H ( ˆ qu′ , p (y | u′ )) , the optimal model parameters corresponding to the new training set become:

In Equation 17, ˆ θ minimizes the loss function on the training set, which means the gradient w.r.t ˆ θ is 0:

Using a Taylor-series approximation at θ∗ , Equation 18 can be rewritten as:

which gives (please refer to Appendix B for details):

With the help of the chain rule dL dϵ = dL dθ · dθ dϵ , the importance of an unlabeled example can be estimated:

Equation 21 is used to compute scoreθ(u) for each unlabeled example. The unlabeled examples with the highest score are preserved to build the core set, and others will be simply dropped. In our implementation, the INFUSE score is calculated batch-wise to reduce the computation overhead. Besides, we use the identity matrix to approximate the inverse Hessian H-1 θ [Luketina et al., 2016] for efficiency. The last problem is how to compute ∇θL (V, θ) when the ground-truth label of examples in V is unavailable in training. To address this, we propose a feature-level mixup to build a support set S. Then, the gradient on the validation set is approximated by L (S, θ). Please refer to Appendix C for details.

5 Experiments

5.1 Experiment Settings

We evaluate the effectiveness of our method on standard semi-supervised learning (SSL) datasets: CIFAR10/100 [Krizhevsky et al., 2009], SVHN [Netzer et al., 2011], STL-10 [Coates et al., 2011]. We follow the commonly used SSL setting [Sohn et al., 2020] for model training. The keep ratio k controls the size of the core set. For example, with k = 10%, the core set size is 10%× |U|, and the total training steps become 10% of the original iterations.

The model is trained under the most commonly used SSL setting [Sohn et al., 2020]. The total number of iterations is 220 (segmented into 1024 epochs) and batch-size of labeled/unlabeled data is 64/448. We use SGD to optimize the parameters. The learning rate is initially set as η0 = 0.03 with a cosine learning rate decay schedule as η = η0 cos ( 7πk 16K ), where k is the current iteration and K is the total iterations.

As for VCC, the size of random noise z is set as 16 for best performance. To reduce the computation overhead, the encoder qϕ and decoder pθ are MLPs with 2 hidden layers (with dimensions 256 and 64). λV CC is set as 2.0.

In INFUSE, the core set is updated for every 40 epochs, and the total number of iterations is adjusted with the keep ratio k. Take k = 10% for example, the amount of examples in core set is 10%× |U| and the total steps is 10%× 220 .

5.2 Main Results

In this section, we present the effectiveness of VCC and INFUSE individually and then combine them to achieve more efficient and accurate pseudo label selection in SSL.

As mentioned earlier, VCC is a general confidence calibration plugin, allowing flexible combinations with existing SSL methods. In our experiments, we choose popular methods like FixMatch [Sohn et al., 2020], FlexMatch [Zhang et al., 2021], and SimMatch [Zheng et al., 2022] as the basic modules to build VCC-FixMatch, VCC-FlexMatch, and VCCSimMatch. The reported values are the mean and standard deviation of three independent trials for each setting, as shown in Table 1. All three baseline methods (FixMatch, FlexMatch, SimMatch) exhibit accuracy improvements when combined with VCC for confidence calibration. Particularly, the improvements with VCC are more pronounced when the amount of labeled examples is small. For instance, on CIFAR-100 with only 400 labeled examples, VCC-FlexMatch reduces the error rate from 46.47% to 43.31% (-3.16%). A similar boost is observed on the STL-10 dataset, as shown in Table 2, where VCC reduces the error rate of FixMatch by 5.34% (from 35.97% to 30.63%) with only 40 labels.

To further understand the source of the accuracy improvement with VCC, we compute the calibration error of different methods. As shown in Table 3, both VCC-FixMatch and VCC-FlexMatch achieve lower calibration errors compared to the baseline methods across various settings. VCCSimMatch also achieves lower Expected Calibration Error (ECE) and Average Calibration Error (ACE) metrics when only 400 labeled examples are available. However, the Maximum Calibration Error (MCE) metric deteriorates, attributed to MCE considering the worst-calibrated bucket and introducing some fluctuations. Under the setting of using 10,000 labeled examples, the results of VCC-SimMatch and SimMatch are very close. This is partly because a larger number of labeled examples can naturally improve the model's performance and reduce the calibration error. Additionally, SimMatch uses instance similarity for rescaling the confidence score, which may reduce the benefits brought by VCC.

The results of INFUSE and other core set selection methods (e.g., RETRIEVE [Killamsetty et al., 2021b]) are shown in Table 4. On CIFAR-10 dataset, INFUSE achieves a lower error rate (6.29%) using only 10% of the examples, indicating the redundancy of original unlabeled data and underscoring the significance of core set selection in SSL. With increasing keep ratio, the gap between INFUSE and the non-pruned setting becomes smaller. For example, on the CIFAR-100 dataset with 2500 labeled data and a keep ratio of 40%, INFUSE achieves an error rate of 26.47%, while the baseline is 26.49%. Compared with other core set selection methods, INFUSE also achieves lower error rates in most settings.

The results demonstrate the effectiveness of VCC and INFUSE individually. By combining them, we propose the VCC-INFUSE method, with results shown in Table 5. VCCINFUSE achieves a better trade-off between model performance and computation costs. Compared to FlexMatch, VCC-INFUSE-FlexMatch not only reduces the error rate from 26.49% to 25.41% but also decreases the training time from 223.96 GPU Hours to 115.47 GPU Hours (-48.44%).

5.3 Ablation Study

We utilize view consistency, temporal consistency, and ensemble consistency for estimating ˜ r. These three consistency scores are designed to reflect the stability of predictions from different perspectives. To analyze their contributions, we conduct an ablation study, and the result is shown in Table 7. As observed, each consistency score contributes to the estimation of a more accurate ˜ r, resulting in a lower error rate.

5.4 Effectiveness of VCC

In VCC, we initially approximate calibrated confidence to obtain ˜ ru. Subsequently, we use a Variational Autoencoder (VAE) to reconstruct it, yielding ru, which is employed in pseudo label selection. The objective of reconstruction is to mitigate the randomness associated with statistical approximation. To demonstrate its necessity, we conduct an ablation study. As shown in Table 8, VCC with reconstruction further reduces the error rate by 0.50%.

5.5 VCC vs. Other CalibrationMethods

While most calibration methods designed for fullysupervised settings may not be directly suitable for SSL, pseudo labels can be used to approximate ground truth. We choose Ensemble-TS [Zhang et al., 2020] and MMCE [Kumar et al., 2018] as baselines to compare with VCC. As depicted in Table 6, MMCE exhibits the highest error rate (28.44%). The reason is that MMCE directly employs pseudo labels to calculate calibration regularization, which may introduce noise due to incorrect pseudo labels. In contrast, Ensemble-TS, using pseudo labels to search for optimal parameter scaling, alleviates the issue to some extent (ER=26.36%). In comparison, VCC achieves the lowest error rate of 25.26% and the best calibration performance.

6 Conclusion

In this study, we addressed the challenges associated with leveraging large-scale unlabeled data in SSL and proposed two novel methods, VCC and INFUSE, to enhance the effectiveness and efficiency of data selection. As a versatile plugin, VCC significantly improves the accuracy of FixMatch, FlexMatch, and SimMatch across multiple datasets. Simultaneously, INFUSE achieves competitive or even lower error rates with partial unlabeled data. By combining the two methods, VCC-INFUSE achieves a lower error rate with less computational overhead. Future work will involve extending VCC-INFUSE to various SSL tasks, such as object detection and segmentation, to assess its generalization.

Acknowledgements

This work was supported by National Key R&D Program of China (No.2018AAA0100300) and the funding of China Tower.

Contribution Statement

Shijie Fang and Qianhan Feng contribute equally to this paper. Tong Lin is the corresponding author. The work was done during Shijie Fang's study at Peking University.