Towards Regularized Mixture of Predictions for Class-Imbalanced Semi-Supervised Facial Expression Recognition

1 Introduction

Facial expressions, a fundamental form of non-verbal communication, are crucial for human-to-human and humancomputer interactions [Li and Deng, 2022]. Recently, some semi-supervised facial expression recognition (SSFER) algorithms [Florea et al., 2020; Li et al., 2022; Roy and Etemad, 2024] have been proposed to improve model performance by assigning pseudo-labels to a large number of confident unlabeled samples. A common assumption is that the class distribution of the constructed semi-supervised facial expression datasets is balanced, meaning the number of facial expression samples in each class is nearly equal. However, this assumption may be unsatisfactory in realistic scenarios, as the class distributions in most facial expression datasets are imbalanced [Li et al., 2021; Zhang et al., 2023].

∗Corresponding author

It is well known that the model trained on class-imbalanced labeled samples is biased towards the majority classes [Menon et al., 2021], for example, Happiness and Neutral in FER. This issue can be further exacerbated for existing semi-supervised learning (SSL) methods, since the pseudolabels assigned to confident unlabeled samples may also be biased, leading to an even more severely imbalanced training set. However, traditional class-imbalanced learning methods, usually designed for labeled samples, cannot be easily combined with SSL methods [Lee et al., 2021]. Therefore, to better facilitate real-world scenarios, we aim to design an SSFER

anced samples, which has not yet been thoroughly explored.

Recently, some SSL methods [Lee et al., 2021; Guo and Li, 2022; Ma et al., 2024] combat the biased pseudo-labels caused by class imbalance at the level of linear prediction, but they often overlook the impact of feature similarity. This oversight becomes more exacerbated in FER, where the main challenge is significant intra-class variations and inter-class similarities at the feature level [Li et al., 2019; Ruan et al., 2021]. In this context, feature similarity and linear prediction are crucial for generating high-quality pseudolabels. To this end, we first examine the effects of linear prediction and feature similarity on pseudo-label quality. There are three sources that guide the model to pseudo-label unlabeled samples: one from a linear classifier, one from a feature classifier (e.g., a collection of class centers [Wen et al.,

2016]), and one from the combination of both classifiers. As shown in Figure 1(b), we observe that the mixed manner consistently enhances the quality of pseudo-labels with higher F1 scores than others. This observation provides excellent insights into learning unlabeled facial expression samples by considering feature and linear prediction levels.

In this paper, we propose a simple yet effective method for class-imbalanced SSFER with a regularized mixture of predictions, which performs a reliable way to generate pseudolabels for imbalanced samples. Specifically, we first use a feature classifier to learn a feature prediction, reflecting the feature similarity between a facial expression feature and the class centers [Wen et al., 2016]. Meanwhile, a linear classifier projects the feature to a linear prediction. We then merge the linear and feature ones into a mixture of predictions to generate a high-quality pseudo-label for an unlabeled sample. However, merely combining these two predictions cannot fully mitigate the bias from the class imbalance issue. To overcome this challenge, inspired by neural collapse [Papyan et al., 2020], we introduce a class regularization term into the feature classifier that constrains the geometry of the feature distribution. This ensures the discriminative power of facial expression features with an explicit inter-class boundary. Furthermore, our method is effective in classifying facial expressions during the inference stage. Overall, the main contributions of this work can be summarized as follows:

• To the best of our knowledge, this is the first practical solution for class-imbalanced semi-supervised facial expression recognition. As a plug-and-play solution, it exhibits strong transferability across various SSL and SSFER methods like FixMatch and Ada-CM.
• We design a mixture of predictions to generate highquality pseudo-labels and enhance inference performance. In addition, we introduce a class regularization term to improve the discriminative power of features.
• Extensive experiments on four challenging datasets demonstrate the superiority of our method under various imbalanced settings, compared to the state-of-the-art class-imbalanced SSL algorithms.

2 RelatedWork

Facial Expression Recognition. An underlying objective of Facial Expression Recognition (FER) is to extract discriminative facial expression features along with a linear classifier for predicted class distributions. In deep learning, numerous FER methods have been proposed under this paradigm [Li et al., 2024]. Typically, Li et al. [2019] proposed classifying facial expressions with attention mechanism from partiallyoccluded faces. Wang et al. [2020] proposed a suppressing method to overcome the uncertainty issue in FER. Xue et al. [2021] applied the Vision Transformers to explore the relation-aware facial expression features. Wu et al. [2023] leveraged facial landmarks to reduce the impact of noisy supervision. Until recently, the issue of class imbalance has also been widely explored in FER. For example, Li et al. [2021] designed an adaptive regular loss to re-weight the importance of different facial expression categories. Zeng et al. [2022]

leveraged large-scale unlabeled images to mitigate the data bias from class-imbalanced facial expression samples. Zhang et al. [2023] proposed to extract extra information related to the minority category from all training samples.

While most of these methods have achieved superior performance in a fully-supervised manner, they heavily consume a large number of labeled samples for the model's training. Unlike them, several methods have leveraged unlabeled samples to explore FER performance in a semi-supervised manner [Jiang and Deng, 2023]. Particularly, Florea et al. [2020] proposed to predict artificial labels of unlabeled samples by center embeddings. Li et al. [2022] designed an adaptive confidence margin to fully learn unlabeled samples. Du et al. [2023] further considered the label ambiguity issue among labeled samples and enhanced the learning on unlabeled samples. However, all these methods are designed under the balanced class distribution. To the best of our knowledge, we are the first to achieve SSFER with class-imbalanced facial expression samples.

Class-Imbalanced Semi-Supervised Learning. Recent semi-supervised learning (SSL) has a long history of research. A typical pipeline in SSL generates pseudo-labels for confident unlabeled samples using the model's outputs, which are used for supervised learning. Take the recent FixMatch [Sohn et al., 2020] as an example. Specifically, it generated pseudo-labels for unlabeled samples with highconfidence predictions above a pre-defined threshold, then leveraged their weak and strong augmentations to achieve consistency regularization. While these methods have seen success in the balanced class scenarios, they fail to improve the biased model's performance when encountering classimbalanced training samples.

To alleviate the above practical problem, class-imbalanced semi-supervised learning (CISSL) methods have garnered increasing attention in recent years [Oh et al., 2022; Wei and Gan, 2023; Lee and Kim, 2024]. For example, Kim et al. [2020] proposed softly refining the biased pseudo-labels by solving a convex optimization problem. Wei et al. [2021] selected more pseudo-labeled samples from minority categories to retrain the baseline SSL model. Lee et al. [2021] introduced an auxiliary balanced classifier in FixMatch to mitigate the class imbalance. Unlike the fixed confidence thresholding, Guo et al. [2022] selected pseudo-labeled samples based on the adaptive thresholds for different categories. Yu et al. [2023] introduced the concept of energy scores from out-ofdistribution detection to generate pseudo-labels for unlabeled samples, which can address the drawbacks of previous confidence scores. Very recently, Ma et al. [2024] proposed to model various class distributions by multiple experts trained with different logit adjustments. These methods mainly focus on the predictions from linear classifiers but ignore the importance of feature similarity. In contrast, we discover the solution of a regularized mixture of predictions for pseudolabeling class-imbalanced unlabeled samples.

3 Preliminary

The Neural Collapse (NC) [Papyan et al., 2020] reveals an intriguing phenomenon that during the terminal phase of training a deep classification model on the balanced dataset, the learned features of the same class will converge to their class centers. Meanwhile, these class centers will exhibit a simplex equiangular tight frame (ETF). The ETF structure can maximize the inter-class difference.

Definition 1 (Simplex Equiangular Tight Frame) For a Kclass classification problem, a simplex ETF is a collection of vectors mk ∈ Rd , k = 1, 2, ..., K if:

where M = [m1, ...,mK] ∈ Rd×K , d is the dimension of the vector, U ∈ Rd×K is an orthogonal matrix satisfying U⊤U = IK, IK is an identity matrix, and 1K ∈ RK is a vector of all ones.

Then, the vectors in a simplex ETF construct an optimal geometry for image classification, which have an equal l2 norm and the same pair-wise angle, i.e.,

where δk,t equals to 1 when k = t and 0 otherwise. The pairwise angle - 1 K-1 is the maximal equiangular separation of K vectors in the feature space Rd [Yang et al., 2022].

Based on the above definition, two important geometry properties derived from the neural collapse phenomenon can be formally summarized as: (NC1) Variability collapse. Intra-class variability of the last-layer features collapses to zero during the terminal phase of training, i.e., ||xi -hk|| = 0, where xi is i-th feature from class k, and hk is the center of class k; (NC2) Convergence to a simplex ETF. The normalized class centers, i.e., ˜hk = (hk-hG)/||hk-hG||, converge to a simplex ETF satisfying Eq. (2), where hG = Avgk{hk} is the global mean of all centers.

4 Method

4.1 ProblemFormulation

For a K-class imbalanced semi-supervised FER task, we have both a batch of labeled samples X = {(xi, yi)}Ni=1 and unlabeled samples U = {xuj }µN j=1 , where µ is a hyper-parameter to determine the relative ratio of X and U, and xi, xuj ∈ Rd are facial expression features for labeled and unlabeled samples, respectively. For the i-th labeled feature xi, it is associated with a ground-truth label yi ∈ {1, 2, ...,K}. Let Nk denote the number of labeled samples in class k, we assume that the classes are sorted in descending order, i.e. N1 ≥ N2 ≥ ... ≥ NK. The degree of class imbalance is defined as the imbalance ratio γ = N1 NK . Similarly, the same assumption for unlabeled samples exists in the CISSL setting. Considering that the training samples from existing facial expression datasets are inherently class-imbalanced [Zhang et al., 2023], we therefore preserve the natural effect of class imbalance on unlabeled samples.

A basic goal of class-imbalanced semi-supervised FER is to train a linear classifier Fl parameterized by θl using X and U. Specifically, let pl(y | x) = Fl(x; θl) be the linear prediction produced by the linear classifier for input x, there is an objective similar to FixMatch [Sohn et al., 2020], consisting of a supervised term and an unsupervised term:

where H is the cross-entropy loss, 1(·) is the indicator function, and τ is a threshold to decide whether or not to retain the pseudo-label. α(·) and A(·) denote the weakly-augmented operation and strongly-augmented operation, respectively. To obtain the pseudo-label ̂ yj = argmax k (yj), it is general to

compute the model's prediction for the weakly-augmented version of the unlabeled sample: yj = pl(y | α(xuj ))1 . In this regard, our work aims to enhance pseudo-label quality at the feature level.

4.2 The ProposedMethod

In class-imbalanced semi-supervised facial expression recognition, the core challenge is how to generate high-quality pseudo-labels for imbalanced facial expression samples. To address this issue, we propose a novel method, named Regularized Mixture of Predictions (ReMoP). In the following section, we will provide an overview of the proposed method and elaborate on key technologies.

Overview. Figure 2 depicts the framework of our proposed ReMoP method. Specifically, in each forward pass, a labeled sample and an unlabeled sample are input into the same encoder for their features using weakly-augmented (WA) and strongly-augmented (SA) operations. Then, we follow the standard SSL learner to project the WA labeled feature and the SA unlabeled feature to the linear predictions using a linear classifier. To generate high-quality pseudo-labels for unlabeled samples, we project the WA unlabeled feature to a mixture of predictions pm(y | α(xuj )), which combines a feature prediction from the feature classifier and a linear prediction from the linear classifier. Meanwhile, we select all labeled features and some unlabeled features with confidence scores higher than the threshold τ, to update the feature classifier. Finally, a class regularization term Lreg is introduced to constrain the geometry of the feature distribution.

Learning a mixture of predictions. In this work, we combine the linear prediction and the feature prediction (i.e., feature similarity). Instead of linear pseudo-labels in the existing SSL learner, we use a mixture of predictions to generate high-quality pseudo-labels for unlabeled samples. Specifically, given a linear classifier Fl, we first learn a linear prediction pl(y | α(xuj )) for a WA unlabeled feature α(xuj ). To execute the mixture of predictions, we initialize a feature classifier H ∈ Rd×K by

where hk ∈ Rd×1 denotes the center for the k-th class. Let ps(y | α(xuj )) be the feature prediction produced by the feature classifier, we normalize the feature and the k-th center hk for their similarity score as

Then, we combine the above two predictions and obtain a mixture of predictions as

1 In SSL, it is popular to generate a pseudo-label using the prediction on the weakly-augmented unlabeled sample, which is used to match the prediction on the strongly-augmented version.

Algorithm 1 ReMoP's main learning algorithm.

Input: A batch of labeled data X = {(xi, yi)}Ni=1 and unlabeled data U = {xuj }µN j=1 , number of epochs Emax, number of training iterations tmax, and the initial parameters θl and H.

Output: Updated parameters θl and H.

• 1: // Training
• 2: for e = 1, 2, ...,Emax do
• 3: for t = 1, 2, ..., tmax do
• 4: // Exploring labeled and unlabeled samples
• 5: Learn the linear prediction pl(y | α(xi)) for labeled sample xi using Fl.
• 6: Learn the mixture of predictions pm(y | α(xuj )) for unlabeled sample xuj by Eq. (6).
• 7: Generate the pseudo-label ̂ ye j by Eq. (7).
• 8: Update model parameters by Eq. (3).
• 9: // Learning a feature classifier
• 10: Update class centers using labeled features and confident unlabeled features by Lc.
• 11: Constrain the geometry of centers by Eq. (8).
• 12: end for
• 13: end for
• 14: // Testing
• 15: Deploy classifiers Fl and H for the mixture of predictions.

where β is a hyper-parameter to balance two predictions. Finally, we use the mixture of predictions to obtain an enhanced pseudo-label:

which is used to replace the linear pseudo-label for the unsupervised term in Eq. (3).

Class regularization term. As previously mentioned, the learning behavior of a classification model on balanced datasets is revealed by a neural collapse phenomenon [Papyan et al., 2020]. To be specific, when the model achieves zero training error rate, the learned features will collapse to their class centers, forming an interesting geometric structure (i.e., a simplex ETF) after being globally centered.

However, the model cannot exhibit the ideal structure on imbalanced datasets, resulting in the features of minority classes becoming indistinguishable [Fang et al., 2021]. To this end, we claim that the cosine similarity of any pairs of globally-centered vectors from H should be close to -1/(K - 1) in the imbalanced setting. In this way, we can maximally separate inter-class features. Specifically, we first update class centers [Wen et al., 2016] using labeled samples and confident unlabeled samples via the center loss Lc = 12 ∑ ||x -hy||22 , where hy denotes the y-th center related to the feature x. Then, we introduce a regularization term Lreg into the mixture of predictions to further mitigate the data bias from the class imbalance:

In summary, our method is optimized in an end-to-end process. It is flexible to be integrated into any standard SSL and SSFER frameworks. Finally, we have the following loss function for training:

where λ1 and λ2 are hyper-parameters to balance three terms' intensity. Note that Lbase takes the enhanced pseudo-labels in Eq. (7) as inputs.

Moreover, unlike the linear prediction from a linear classifier used in existing FER pipelines, we merge it to the feature similarity for a mixture of predictions during the testing stage. Specifically, for a testing feature xt , we deploy two classifiers Fl and H for the mixture of predictions pm(y | xt ). The whole progress of the proposed method is summarized in Algorithm 1.

5 Experiments

5.1 Datasets

We conduct experiments on four public facial expression datasets, including RAF-DB [Li and Deng, 2019], FERPlus [Barsoum et al., 2016], CK+ [Lucey et al., 2010], and AffectNet [Mollahosseini et al., 2017]. RAF-DB is a large-scale facial expression dataset with 29,672 real-world facial images, which are labeled by about 40 annotators into a singlelabel subset and a two-tab subset. In our experiments, we use the single-label subset with 12,271 training images and 3,068 testing images, including seven basic facial expression categories (i.e., surprise, fear, disgust, happiness, sadness, anger, and neutral). FERPlus provides the new eight-class labels (i.e., seven basic categories and contempt) created by 10 crowd-sourced annotators. It consists of 28,709 training images, 3,589 validation images, and 3,589 testing images. CK+ consists of 593 video sequences from 123 subjects. For a fair comparison, we follow [Li et al., 2022] to select each sequence's first frame and the last frame as the neutral face and the targeted facial expression, containing 636 facial images with seven basic categories. Different from the first three datasets, AffectNet is by far the largest in-the-wild facial expression dataset, containing more than 1M facial images but a large number of noisy labels. In our experiments, we choose about 287,651 manually annotated images with seven basic categories and the contempt category as inter-dataset unlabeled samples to evaluate the CISSL performance.

Following the setting in CISSL [Guo and Li, 2022], we use the imbalance ratio γ with the given N1 to construct the class-imbalanced training set. Specifically, we set the number of labeled samples in class k as Nk = N1 · γ- k-1 K-1 for 1 < k ≤ K. In this work, we design various combinations of γ and N1. Considering the imbalanced testing class distribution in RAF-DB, FERPlus, and CK+, we report the overall accuracy and the mean accuracy across all categories by default. Unless otherwise specified, we conduct experiments three times using different random seeds to obtain the mean and the standard deviation.

5.2 Implementation Details

We implement all experiments using the PyTorch toolbox with one NVIDIA A100 GPU. For the basic encoder, we use the ResNet-18 [He et al., 2016] pre-trained on the MS-Celeb1M face recognition dataset [Guo et al., 2016] for learning facial expression features. Besides, we use MTCNN [Zhang et al., 2016] to align and resize facial images to 224×224 pixels.

During training, we follow [Li et al., 2022] to use RandomCrop and RandomHorizontalFlip as the weak augmentation strategy, and add RandAugment [Cubuk et al., 2020] as the strong augmentation strategy for a fair comparison. By default, we use the Adam optimizer. For RAF-DB and CK+, we train the model with a learning rate of 1e - 4, training epoch 100, and batch size 16. For FERPlus, due to a large number of training samples, we train the model with a learning rate of 1e - 4, training epoch 80, and batch size 32. The number of training iterations tmax is set to 1,000 in all experiments. The relative ratio µ is set to 1 except for AffectNet as 5. The hyper-parameter β in Eq. (6) is set to 0.5. Following FixMatch [Sohn et al., 2020], the default threshold τ is 0.95. In Eq. (9), the hyper-parameters λ1 and λ2 are set to 1e - 4 and 0.1, respectively.

5.3 Ablation Study

Effect of two modules in ReMoP. We examine the effectiveness of ReMoP in Table 1. Several observations can be summarized as follows: 1) Compared with the Vanilla-based baseline using a linear classifier (row 1), introducing a feature classifier for the mixture of predictions (row 2) consistently improves inference performance; 2) A significant improvement between rows 2 and 3 is achieved after introducing Lreg into H. Since the class regularization term is used to constrain the feature classifier for the geometry of feature distribution, it is reasonable to enhance the discriminative power of facial expression features; 3) The similar improvements (rows 4 to 6) are present in the learning of unlabeled samples as well. These results also verify that the proposed method effectively learns unlabeled facial expression samples at feature and linear prediction levels.

Effect of the mixture of predictions for testing samples. To eliminate the concern that the mixture of predictions is more effective in determining facial expression categories than the linear prediction during the testing stage, we conduct an ablation study to evaluate their difference. As shown in Figure 3, the mixture strategy consistently improves performance in each case. This contributes an important insight for FER to explore the impact of discriminative features.

Effect of varying hyper-parameter β. In Figure 4, we conduct an ablation study on RAF-DB to analyze the effect of different value β ∈ {0.1, 0.3, 0.5, 0.7, 0.9}. The hyperparameter β reflects the intensity of two predictions in the mixture of predictions. From the results, we can observe that when β is too large or too small, the performance decreases as the excessive importance of linear prediction or feature similarity hampers the quality of pseudo-labels. We fix β = 0.5 for all the experiments according to the experimental results. Effect of ReMoP on feature learning. As mentioned above, the proposed ReMoP constrains the geometry of the feature distribution. To verify this, we visualize the t-SNE distribution [Van der Maaten and Hinton, 2008] for facial expression features learned by FixMatch and FixMatch-based ReMoP. As shown in Figure 5, ReMoP helps construct a discriminative feature space, which can boost FER performance.

Effect of ReMoP using different SSL learners. In this work, we claim that the proposed method as a plug-andplay solution can be integrated into existing SSL and SSFER methods. To evaluate this, we conduct experiments using a typical SSL method (FixMatch) and an SSFER method (AdaCM). As shown in Table 2, using ReMoP further improves the performance on overall and mean accuracy in each case, demonstrating the ability of the proposed method to address the class-imbalanced issue in SSFER. In the following, we use FixMatch [Sohn et al., 2020] as the baseline SSL learner.

5.4 Comparison with State-of-the-Art Methods

In this section, based on the widely-used FixMatch method [Sohn et al., 2020], we compare the proposed method with some popular CISSL methods, including ABC [Lee et al., 2021], Adsh [Guo and Li, 2022], InPL [Yu et al., 2023], and CPE [Ma et al., 2024].

Intra-dataset evaluation. In this part, we assume that labeled and unlabeled samples come from the same dataset, that is, they are randomly divided from original training set.

As shown in Table 3, we compare the proposed ReMoP with state-of-the-art CISSL methods on RAF-DB under four different class distributions. From these results, we can see that 1) the proposed method can consistently improve the vanilla method, validating the effectiveness of ReMoP to leverage unlabeled samples for discriminative learning; 2) the proposed method achieves satisfactory performance, outperforming existing methods by a large margin in most cases. For example, given the setting of (N1, γ) = (2000, 150), the proposed ReMoP significantly outperforms the competitive Adsh [Guo and Li, 2022] by 1.80% and 3.34% on overall and average accuracy, respectively.

In addition, Table 4 compares the performance on CK+ and FERPlus under two different class distributions, respectively. We can see that the proposed ReMoP steadily and significantly outperforms the state-of-the-art CISSL methods. For example, ReMoP outperforms CPE [Ma et al., 2024] by 2.62% and 3.17% overall accuracy on FERPlus under two different settings, respectively. These results demonstrate the superiority of the proposed method in learning with classimbalanced samples.

Inter-dataset evaluation. In this part, we assume that labeled and unlabeled samples come from different datasets.

In Table 5, we conduct experiments on RAF-DB and FERPlus. Following the setting [Jiang and Deng, 2023], we treat the training images along with their labels from RAF-DB or FERPlus as labeled samples, and the training images from AffectNet as unlabeled samples. From the table, we can see that the proposed ReMoP significantly outperforms the existing methods by a large margin. Specifically, on the RAFDB dataset, our method achieves 2.15% overall accuracy improvements compared to the competitive CPE [Ma et al., 2024] and outperforms PT [Jiang and Deng, 2023] by 1.6% overall accuracy. Furthermore, we compare the proposed ReMoP with some fully-supervised methods. For example, our method outperforms MEK [Zhang et al., 2023] by 0.52% overall accuracy on RAF-DB.

6 Conclusion

In this work, we address the practical class-imbalanced semisupervised facial expression recognition task, focusing on the quality of pseudo-labels for imbalanced samples in a reliable manner. To this end, we propose the Regularized Mixture of Predictions (ReMoP) method, which integrates feature similarity with linear prediction to create a mixture of predictions. Additionally, we introduce a class regularization term that effectively handles class-imbalanced feature distribution. Through extensive experiments on four facial expression datasets, we demonstrate the efficacy of ReMoP across various challenging imbalanced setups. Our work may further contribute to understanding the importance of feature similarity in classifying facial expressions.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants U22A2096, 62036007, 62376235, and 62306178, in part by Scientific and Technological Innovation Teams in Shaanxi Province under grant 2025RS-CXTD-011, in part by the Shaanxi Province Core Technology Research and Development Project under grant 2024QY2-GJHX-11, in part by the Fundamental Research Funds for the Central Universities under GrantQTZX23042, in part by the GDST Basic Research Fund Nos. 2022A1515011652 and 2024A1515012399, in part by the HKBU Faculty Niche Research Areas No. RC-FNRAIG/22-23/SCI/04, and in part by the HKBU CSD Departmental Incentive Scheme.

Contribution Statement

Hangyu Li and Yixin Zhang contributed equally to this work.