AllMatch: Exploiting All Unlabeled Data for Semi-Supervised Learning

1 Introduction

Semi-supervised learning (SSL) [Zhu, 2005; Rosenberg et al., 2005; Berthelot et al., 2019b; Sohn et al., 2020], a research topic that aims to boost the model's generalization performance by leveraging the potential of unlabeled data, has received extensive attention in recent years. Among the proposed techniques, the combination of pseudo-labeling [Lee and others, 2013; Arazo et al., 2020] and consistency regulation [Sajjadi et al., 2016; Laine and Aila, 2016], as introduced by FixMatch [Sohn et al., 2020], has emerged as a predominant approach. Specifically, FixMatch first assigns a pseudo-label to each unlabeled sample based on the prediction of its weakly augmented view. Subsequently, pseudo- Figure 1: Pilot study for pseudo-label quality on CIFAR-10 with 40 labels. For SoftMatch, pseudo-labels with a confidence lower than µt -σt are assigned close-to-zero weights and thus considered dropped in the analysis. Here, µt/σt denotes the mean/std of the overall confidence on unlabeled data in SoftMatch.

∗Corresponding author

labels exceeding a predefined confidence threshold are used as supervision for corresponding strongly augmented views, while those below the threshold are discarded. To ensure high-quality pseudo-labels, FixMatch uses a high constant threshold throughout training. However, this strategy results in the underutilization of unlabeled data, presenting a central challenge in SSL: making efficient use of unlabeled data.

To address the trade-off between the quality and quantity of pseudo-labels in threshold-based pseudo-labeling, previous studies introduce dynamic threshold strategies that align with the evolving learning status. For example, FlexMatch [Zhang et al., 2021] utilizes the number of confident pseudo-labels to estimate the learning difficulty for each class, subsequently mapping the predefined threshold to class-specific thresholds based on the determined difficulty levels. Additionally, FreeMatch [Wang et al., 2022] leverages the average confidence of unlabeled data to establish a dynamic global threshold. Besides, SoftMatch [Chen et al., 2023] employs a Gaussian function representing the disparity between individual and global confidence to model sample weights, assigning a positive weight to each unlabeled sample. However, samples with confidence significantly lower than the global confidence receive close-to-zero weights, essentially treated as if they were discarded. This characteristic positions SoftMatch as a variant of the threshold-based method. While these algorithms effectively employ pseudo-labels to assess the learning status, biased data sampling and potential interclass similarities can significantly influence the predictions of unlabeled data. Consequently, a pivotal question arises: Can we incorporate additional evidence along with pseudo-labels to achieve a more accurate estimation of the learning status?

While an improved threshold scheme can enhance the utilization of unlabeled data, a portion of unlabeled samples still face exclusion. This raises another question: Can pseudolabels assigned lower confidence provide valuable semantic guidance? To address this concern, we examine the pseudolabel quality of previous algorithms. In the case of CIFAR10 [Krizhevsky et al., 2009] with 40 labeled samples, more than half of the dropped pseudo-labels prove to be correct, as depicted in Figure 1(a). Furthermore, Figure 1(b) illustrates that the top-5 accuracy of the pseudo-labels reaches 100% within just a few thousand iterations. Accordingly, pseudolabels with lower confidence can eliminate false options (e.g., bottom-5 classes) and provide effective supervision signals.

Motivated by the aforementioned questions, this paper introduces AllMatch, a novel SSL model designed to enhance learning status estimation and provide semantic guidance for all unlabeled data. Specifically, AllMatch proposes a classspecific adaptive threshold (CAT) strategy, comprising global estimation and local adjustment steps, to achieve an improved characterization of the model's learning status. The global estimation step, similar to FreeMatch, employs the average confidence of unlabeled data as the global threshold. The ensuing local adjustment step utilizes the classifier weights to estimate the learning status of each class, adaptively decreasing thresholds for classes facing challenges. As illustrated in Figure 4(b, c) and Figure 4(f, g), CAT outperforms previous approaches in terms of the utilization ratio and pseudolabel accuracy of unlabeled samples. Besides, in response to the underutilization of unlabeled data resulting from the exclusion of low-confidence pseudo-labels, AllMatch introduces a binary classification consistency (BCC) regulation strategy to exploit the latent potential within such pseudolabels. In essence, the BCC regulation divides the class space into candidate and negative classes, encouraging consistent candidate-negative division across diverse perturbed views of the same sample to eliminate negative options. The candidate class for each sample corresponds to its top-k predictions, considering the impressive top-k performance of various algorithms. Note that the parameter k is dynamically determined based on varying sample-specific learning status and evolving model performance. As depicted in Figure 4(c, d) and Figure 4(g, h), the BCC regulation effectively identifies candidate classes for unlabeled samples and achieves a 100% utilization ratio for the unlabeled data. Overall, our contributions can be summarized as follows:

1. We revisit existing SSL algorithms and raise two questions: how to develop an effective threshold mechanism and how to utilize the low-confidence pseudo-labels.
2. We propose the class-specific adaptive threshold mechanism, which employs pseudo-labels and classifier weights to estimate global and class-specific learning status respectively.
3. We design the binary classification consistency regulation to provide supervision signals for all unlabeled samples.
4. We conduct experiments on multiple benchmarks, considering both balanced and imbalanced settings. The results indicate that AllMatch achieves state-of-the-art performance.

2 RelatedWork

Consistency regulation represents the fundamental approach in SSL. The primary objective is to ensure consistent predictions across different perturbed views of the same sample. The original approaches [Laine and Aila, 2016; Sajjadi et al., 2016] minimize the L2 distance between the predictions of two randomly augmented views. The Mean Teacher method [Tarvainen and Valpola, 2017] extends this approach by replacing one of the predictions with the output of an exponential moving average (EMA) model. Furthermore, FixMatch [Sohn et al., 2020] combines consistency regulation and pseudo-labeling techniques, establishing an effective SSL paradigm. To avoid similar perturbed views, [Sohn et al., 2020; Xie et al., 2020] use the predictions of weakly augmented samples as pseudo-labels and minimize their divergence from the predictions of the corresponding strongly augmented views. Additionally, these methods use cross-entropy loss instead of the L2 distance for stable training.

To ensure high-quality pseudo-labels, FixMatch [Sohn et al., 2020] utilizes a high constant threshold throughout training to filter out potentially incorrect pseudo-labels. However, this strategy results in the underutilization of unlabeled data. To address this issue, FlexMatch [Zhang et al., 2021] draws inspiration from curriculum learning [Bengio et al., 2009], mapping the predefined threshold to classspecific thresholds based on the learning status of each category. Dash [Xu et al., 2021] defines the threshold based on the loss of labeled data, eliminating the empirical threshold parameter. FreeMatch [Wang et al., 2022] employs the average confidence on unlabeled data as the adaptive global threshold. SoftMatch [Chen et al., 2023] estimates sample weights by a dynamic Gaussian function, maintaining soft margins between unlabeled samples of different confidence levels. Notably, samples with confidence significantly lower than the global confidence are assigned close-to-zero weights in SoftMatch. Accordingly, such samples can be considered dropped, rendering SoftMatch a variant of the thresholdbased method. In addition to the threshold-based algorithms, CoMatch [Li et al., 2021] and SimMatch [Zheng et al., 2022] leverage contrastive loss to impose sample-level constraints on all unlabeled data. In contrast, AllMatch combines the advantages of both threshold-based and contrastive-based methods. The approach introduces CAT, a learning-status-aware threshold strategy, and BCC regulation, the semantic-level supervision for the entire unlabeled set, to maximize the utilization of unlabeled data and enhance the overall performance.

In addition to consistency regulation and pseudo-labeling, entropy-based regulation is another widely adopted strategy. Entropy minimization [Grandvalet and Bengio, 2004] promotes high-confidence predictions during training. Maximizing the entropy of the expectation over all samples [Krause et al., 2010; Arazo et al., 2020; Zhao et al., 2022] introduces the concept of fairness, which encourages the model to predict each class with equal frequency. Specifically, distribution alignment (DA) [Berthelot et al., 2019a] and uniform alignment (UA) [Chen et al., 2023] are prevailing strategies for achieving fairness in SSL, which adjusts the pseudo-label based on the overall predictions on the unlabeled data.

3 Methodology

3.1 Preliminary

We begin by reviewing the widely adopted SSL framework. Let DL = {(xi, yi)}NL i=1 and DU = {ui}NU i=1 represent the labeled and unlabeled datasets, respectively. Here, xi and ui denote the labeled and unlabeled training samples, and yi represents the one-hot label for the labeled sample xi. We denote the prediction of sample x as p(y|x). Given a batch of labeled and unlabeled data, the model is optimized with the objective L = Ls + λuLu. Here, Ls represents the crossentropy loss (H) for the labeled batch of size BL.

Lu indicates the consistency regulation between the prediction of the strongly augmented view Ω(u) and the pseudolabel derived from the corresponding weakly augmented view ω(u). To filter out incorrect pseudo-labels, FixMatch [Sohn et al., 2020] introduces a predefined threshold τ . Specifically, Lu is defined as follows.

Here, ˜ pi is the abbreviation for DA(p(y|ω(ui))), where DA indicates the distribution alignment strategy [Berthelot et al., 2019a]. ˆ pi represents the one-hot pseudo-label obtained from argmax( ˜ pi). Moreover, qi is the abbreviation for p(y|Ω(ui)). Lastly, BU corresponds to the batch size of unlabeled data.

3.2 Class-Specific Adapative Threshold

Previous studies [Zhang et al., 2021; Wang et al., 2022] have demonstrated that the threshold should be aligned with the evolving learning status of the model. To achieve this, these approaches leverage predictions on unlabeled data to establish the dynamic threshold. In this paper, we unveil the ability of the classifier weights to differentiate the learning status of each class. By combining pseudo-labels and classifier weights, we introduce a class-specific adaptive threshold (CAT) mechanism. As depicted in Figure 2, CAT comprises the global estimation and local adjustment steps. The following parts provide a detailed description of these two steps.

Global Estimation. The global estimation step learns from FreeMatch [Wang et al., 2022] and evaluates the overall learning status of the model. Given that deep neural networks tend to prioritize fitting easier samples before memorizing harder and noisier ones, a lower threshold is necessary during early training stages to incorporate more correct pseudolabels. Conversely, as training progresses, a higher threshold is required to filter out incorrect pseudo-labels. Given that the cross-entropy loss encourages confident predictions, the average confidence of the unlabeled set captures information from all unlabeled data and steadily increases throughout training, thereby reflecting the overall learning status. However, making predictions for the entire unlabeled set at each time step incurs significant computational costs. Accordingly, we employ the mean confidence of the current batch as an estimation and update it using exponential moving average (EMA). Specifically, the global learning status estimation at t-th iteration, denoted as τt, can be computed as follows:

Here, pi represents p(y|ω(ui)), m denotes the momentum decay, and C corresponds to the number of classes.

Local Adjustment. Due to the inherent variations in learning difficulty among different classes and the stochastic nature of parameter initialization, the model's learning status varies across categories. To address this issue, we introduce the local adjustment step, which makes the model pay more attention to the underfitting classes by decreasing their thresholds. Specifically, our study reveals that the L2 norm of the classifier weights provides insights into class-specific learning status. The reasons are explained as follows.

Firstly, let M = G ◦ F denote the model, where F and G correspond to the encoder and the one-layer classifier, respectively. Given the feature vector f = F(u) ∈ Rd of the unlabeled sample u, the predicted logits z ∈ RC can be computed as z = G(f) = fW T , where W ∈ RC×d denotes the weight matrix of the classifier. The bias term is dropped as it typically has a negligible impact on the results. Consequently, the logits of class c can be expressed as zc = ||f || · ||Wc|| · cos(θ), where || · || denotes the L2 norm, and Wc corresponds to the weight of class c. Accordingly, the model tends to produce logits with larger absolute values for categories with larger weight norms, implying that classes with larger weight norms exhibit a preferred learning status.

Secondly, [Kang et al., 2019] reveals a positive correlation between the weight norm ||Wc|| and nc, the number of samples in class c. Given the abundant unlabeled data and limited labeled data, nc can be approximated as the number of unlabeled samples classified into class c with confidence scores surpassing the threshold. Hence, a larger weight norm suggests that more samples are confidently categorized into the class, signifying a better learning status.

According to the above analysis, the L2 norm of classifier weights characterizes the learning status of each class. Consequently, the local adjustment step leverages this indicator to establish the mapping from the global threshold to classspecific thresholds. Specifically, we linearly scale the threshold for each class based on the deviation of its learning status from the optimal learning status. As such, the threshold for class c at t-th iteration, denoted as ρt(c), can be computed as follows.

Moreover, to ensure a stable estimation of the learning status, we employ the classifier weights obtained from the EMA model. Notably, in contrast to FlexMatch, which maintains an additional list for recording the selected pseudo-label of each sample, the proposed CAT refrains from storing any samplespecific information during training. This eliminates the indexing budget concerns on large-scale datasets. With CAT incorporated, the mask for unlabeled samples in Lu can be expressed as follows.

3.3 Binary Classification Consistency Regulation

While the proposed class-specific adaptive threshold alleviates the underutilization of unlabeled data, a substantial number of pseudo-labels continue to be discarded. As illustrated in Figure 1, in the case of CIFAR-10 with 40 labeled samples, the top-5 accuracy of pseudo-labels effortlessly achieves 100% regardless of the adopted algorithm. In other words, pseudo-labels assigned lower confidence contribute to identifying candidate classes (e.g., top-k predictions) and excluding negative options (e.g., classes not included in top-k predictions). Motivated by these observations and the consistency regulation technique, we propose the binary classification consistency (BCC) regulation, whose overview is shown in Figure 3. In a nutshell, the strategy introduces semantic supervision for all unlabeled data by encouraging consistent candidate-negative division across diverse perturbed views of the same sample. The details are described as follows.

Given the impressive top-k pseudo-label accuracy obtained by numerous algorithms, the BCC regulation adopts the top-k predictions of each unlabeled sample as its candidate classes and the rest as the negative options. Thus, the candidatenegative division is simplified to the selection of the parameter k. Moreover, considering the variations in learning difficulty among different samples and the evolving performance of the model, the candidate-negative division for each sample should be determined based on both individual and global learning status. To achieve this, the BCC regulation first computes sample-specific top-k confidence and the global top-k confidence of the entire unlabeled set. Specifically, let pki denote the top-k probability of sample ui, and µkt represent the global top-k probability at t-th iteration. The global top-k confidence can be estimated by the exponential moving average (EMA) of the average top-k confidence at each time step.

Here, c1, . . . , ck represent the k classes assigned the highest probability in pi. With the global top-k confidence determined, the number of candidate classes for each unlabeled sample is defined as the minimum value that makes individual top-k confidence higher than global top-k confidence. Particularly, the candidate class for confident unlabeled samples is defined as the pseudo-label. Accordingly, the number of candidate classes ki for sample ui can be expressed as follows.

where K is the upper bound for the number of candidate classes to prevent trivial candidate-negative division. With the division obtained, the candidate and negative probabilities for the weakly (bωi ) and strongly (bΩi ) perturbed views of the unlabeled sample ui can be calculated as follows.

Here, c1, . . . , cki represents the ki classes assigned the highest probability in ˜ pi. Finally, the BCC regulation for a batch of unlabeled data can be calculated as follows:

3.4 Overall Objective

The overall objective of AllMatch is defined as the weighted sum of all semantic-level supervision.

λu and λb denote the weights to balance different supervision signals. For all experiments, we set both λu and λb to 1.0.

4 Experiments

4.1 Balanced Semi-Supervised Learning

Settings. For balanced image classification, we conduct experiments on CIFAR-10/100 [Krizhevsky et al., 2009], SVHN [Netzer et al., 2011], STL-10 [Coates et al., 2011], and ImageNet [Deng et al., 2009] with various numbers of labeled data, where the class distribution of the labeled data is balanced. To ensure fair comparisons, we employ the unified codebase TorchSSL [Zhang et al., 2021] to evaluate all methods. Regarding the backbone architecture, we follow previous studies and use specific models for different datasets: WRN-28-2 [Zagoruyko and Komodakis, 2016] for CIFAR-10 and SVHN, WRN-28-8 for CIFAR-100, WRN-37-2 [Zhou et al., 2020] for STL-10, and ResNet-50 [He et al., 2016] for ImageNet. The batch sizes BL and BU are set to 128 and 128 for ImageNet and 64 and 448 for the remaining datasets. AllMatch is trained using the SGD optimizer with an initial learning rate of 0.03 and a momentum decay of 0.9. The learning rate is adjusted by a cosine decay scheduler over a total of 220 iterations. We set m to 0.999 and generate the EMA model with a momentum decay of 0.999 for inference. The upper bound K is set to 20 for ImageNet and 10 for the other datasets. For SVHN, CIFAR-10 with 10 labels, and STL-10 with 40 labels, we constrain the threshold within the range of [0.9, 1.0] to prevent overfitting noisy pseudo-labels in the early training stages. To account for randomness, we repeat each experiment three times and report the mean and standard deviation of the top-1 accuracy.

Performance. Table 1 presents the top-1 accuracy on CIFAR-10/100, SVHN, and STL-10 with various numbers of labeled samples. The performance on ImageNet is reported in Table 2. The experimental results demonstrate that AllMatch achieves state-of-the-art performance on all datasets, except for CIFAR-100 with 10000 labels and SVHN with 40 labels, where our model's performance remains comparable to the best competitor. Moreover, regarding CIFAR100, AllMatch outperforms ReMixMatch when only 400 or 2500 labels are available, while the latter achieves better performance when 10000 labels are available. The competitive results obtained by ReMixMatch mainly stem from the Mixup technique [Zhang et al., 2017] and the additional selfsupervised learning part. Furthermore, AllMatch exhibits substantial advantages over previous algorithms when dealing with extremely limited labeled samples. Specifically, the approach surpasses the second-best counterpart by 1.87% on CIFAR-10 with 10 labels, 0.66% on CIFAR-100 with 400 labels, and 2.86% on STL-10 with 40 labels. Particularly, STL-10 poses significant challenges due to its large unlabeled set that comprises 100k images. Accordingly, the impressive improvement obtained on STL-10 highlights the potential of AllMatch to be deployed in real-world applications.

4.2 Imbalanced Semi-Supervised Learning

Settings. We evaluate AllMatch in the context of imbalanced SSL, where both labeled and unlabeled data exhibit a long-tailed distribution. All experiments are conducted on the TorchSSL codebase. Following prior studies [Lee et al., 2021; Oh et al., 2022; Wei et al., 2021; Lai et al., 2022; Fan et al., 2022; Chen et al., 2023], we generate the labeled and unlabeled sets using the configurations of Nc = N1 · γ- c-1 C-1 andMc =M1 ·γ- c-1 C-1 . Specifically, for CIFAR10-LT, we set N1 to 1500, M1 to 3000, and γ to range from 50 to 150. For CIFAR-100-LT, we set N1 to 150, M1 to 300, and γ to range from 20 to 100. In all experiments, we employ WRN-28-2 as the backbone and utilize the Adam optimizer [Kingma and Ba, 2014] with the weight decay of 4e-5.

The batch sizes BL and BU are set to 64 and 128, respectively. The learning rate is initially set to 2e-3 and adjusted by a cosine decay scheduler during training. We repeat each experiment three times and report the overall performance.

Performance. In the context of imbalanced SSL, we compare AllMatch with several strong baselines, including FixMatch, FlexMatch, SoftMatch, and FreeMatch. The results in Table 3 demonstrate that AllMatch achieves state-of-theart performance on all benchmarks. It is particularly noteworthy that AllMatch outperforms the second-best approach by 1.69% and 1.65% at γ=100 and γ=150 on CIFAR-10-LT, respectively, highlighting its robustness in handling significant imbalances. The consistently impressive performance observed in imbalanced SSL suggests that AllMatch can effectively address real-world challenges.

4.3 Ablation Study

Component analysis. We conduct an ablation study on four challenging datasets: CIFAR-10 with 10 labels, CIFAR-

100 with 400 labels, STL-10 with 40 labels, and CIFAR-10LT with an imbalance ratio of 150. For simplicity, we refer to the performance on the four benchmarks as (a, b, c, d) in subsequent analysis. As shown in Table 4, the global estimation step in CAT (line 2) promotes the performance by (7.35%, 8.00%, 16.45%, 1.20%) compared to the baseline model in line 1. The significant improvement highlights the crucial role of aligning the threshold with the model's global learning status. Furthermore, the local adjustment step in CAT (line 3) yields additional gains of (3.22%, 1.46%, 6.95%, 1.90%), suggesting that it effectively captures class-specific learning difficulties and facilitates the learning of classes facing challenges. Additionally, the BCC regulation enables a 100% utilization ratio of the unlabeled data and achieves the improvement of (2.25%, 0.52%, 0.71%, 0.77%). The substantial improvement observed on CIFAR-10 with 10 labels indicates the potential of the BCC regulation when dealing with extremely limited labeled data. Overall, the results in Table 4 demonstrate the effectiveness of the proposed modules.

Comparative study on threshold strategies. We conduct a comparative analysis of existing threshold mechanisms in two aspects. Firstly, we directly compare the proposed CAT with the threshold strategies adopted in previous models. The results are presented in line 1-4 of Table 5. Secondly, we assess the threshold strategies within the AllMatch framework, i.e., combining existing threshold schemes with the BCC regulation, and the results are provided in line 5-8 of Table 5. From both perspectives, AllMatch outperforms previous models in most cases, indicating the effectiveness of the proposed CAT. Moreover, the BCC regulation further boosts the performance of prior methods, suggesting its impressive compatibility and contribution to eliminating false options.

4.4 Quantitative Analysis

To gain further insights into AllMatch, we present various training indicators on CIFAR-10 with 40 labels and STL-10 with 40 labels, as illustrated in Figure 4. From Figure 4(a) and Figure 4(e), it can be observed that the threshold exhibits the expected behavior, starting with a small value and gradually increasing thereafter. Moreover, AllMatch demonstrates a smoother threshold evolution in contrast to other classspecific threshold-based models, implying a preferred learning status estimation. Additionally, in comparison to previous algorithms, Figure 4(b) and Figure 4(c), as well as Figure 4(f) and Figure 4(g), indicate that AllMatch achieves improved pseudo-label accuracy and a higher utilization ratio for the unlabeled data. Notably, Figure 4(b) and Figure 4(f) demonstrate that previous models consistently suffer from overfitting noisy pseudo-labels in the later training stages, whereas AllMatch successfully mitigates this issue. Furthermore, as depicted in Figure 4(d) and Figure 4(h), the accuracy of the candidate-negative division consistently outperforms pseudolabel accuracy, suggesting that the BCC regulation effectively identifies the candidate classes for all unlabeled data. Overall, the CAT precisely reflects the learning status of the model and the BCC regulation provides accurate supervision signals for all unlabeled samples.

5 Conclusion

This paper revisits prior SSL algorithms, focusing on two crucial questions: how to design an effective threshold mechanism and how to utilize the pseudo-labels assigned lower confidence. To address these challenges, we introduce two strategies named class-specific adaptive threshold (CAT) and binary classification consistency (BCC) regulation. CAT leverages predictions on unlabeled data and classifier weights to establish a threshold mechanism that aligns with the evolving learning status of each class. BCC regulation identifies candidate classes for each unlabeled sample and encourages consistent candidate-negative divisions across diverse perturbed views of the same sample. With these two modules incorporated, the proposed AllMatch maximizes the utilization of unlabeled data and achieves impressive pseudo-label accuracy. We conduct extensive experiments on multiple benchmarks, including both balanced and imbalanced settings. The results demonstrate that AllMatch achieves state-of-the-art performance and is capable of dealing with real-world challenges.