LION: Label Disambiguation for Semi-supervised Facial Expression Recognition with Progressive Negative Learning

1 Introduction

Facial expression is one of the most common displays of human emotion, which plays an important role in interpersonal communications. In the past few decades, with the emergence of large-scale well-labeled facial datasets, e.g., AffectNet [Mollahosseini et al., 2017] and RAF-DB [Li et al., 2017], many automatic facial expression recognition (FER) approaches based on fully supervised deep learning have been proposed and made significant progress in distinguishing facial expressions, including surprise, fear, disgust, happiness, sadness, anger and neutral. However, in the real world, due to the expensive and tedious annotation process, it is extremely difficult to collect a large volume of high-quality labels for a dataset.

Recently, semi-supervised deep facial expression recognition (SS-DFER) methods have been developed to make use of the unlabeled data. For example, Ada-CM [Li et al., 2022] applied the common semi-supervised learning (SSL) techniques, e.g., pseudo labeling and consistency regularization, to the FER task and learned an adaptive confidence margin to fully leverage unlabeled data. Nonetheless, we argue that there are still two main problems lacking consideration in the current SS-DFER methods, which impedes further performance improvement. The first one is the label ambiguity problem in the labeled set. It means that the given labels may mismatch with the facial expressions due to (і) data variations in illumination, resolution, occlusion, and pose, (іі) ambiguity in expressions, (ііі) crowd sourcing, and (iv) low-quality annotations obtained on a search engine. We present some clear and ambiguous examples from RAF-DB dataset in Fig.1. All the images are re-assessed by 35 volunteers. Taking the image annotated with "Surprise" (with red border) for example, the confidence score is only 0.142, and most volunteers think it should be "Fear" instead, which might be the potentially real label (latent truth). Training with these ambiguous data may result in difficulty of convergence and inevitable performance degradation.

The second problem of the current SS-DFER methods concerns the inefficient utilization of unlabeled data. As mentioned above, existing SS-DFER methods typically adopt strategies such as pseudo-labeling and consistency regularization to leverage the unlabeled data. To ensure that reliable information is mined from unlabeled data, these methods usually keep the unlabeled data with high confidence scores predicted by the models, but disregard those data with uncertain predictions. Such naïve operations only explore the value of a fraction of easy unlabeled data, wasting the rich information contained in low-confidence samples.

In this paper, we propose a novel SS-DFER approach with a Label DIsambiguation module and a PrOgressive Negative Learning module, namely LION, to simultaneously address the above two problems, i.e., label ambiguity and inefficient utilization of unlabeled data. For the former, the label disambiguation module is designed to screen out those ambiguously labeled data and re-assign them with more accurate labels. Specifically, if the given label disagrees with the class prediction by the classifier, we define it as an ambiguous label, otherwise a clear label. All the data with clear labels, or clear data, are utilized to derive class prototypes of the seven facial expressions, which guide the model to correct the ambiguous labels. For the ambiguous data, considering the subtle differences among different facial expressions, unlike previous methods that only assign one label to facial images, we maintain a candidate label set with a collection of candidate classes to store potentially real labels (as shown the sets below the facial images in Fig. 1) and update the original labels based on the candidate label set and class prototypes. For the latter problem, we first split the unlabeled data into a reliable set with high-confidence and an unreliable set with low-confidence. All the reliable data will also involve the calculation of class prototypes. As for the unreliable data, since it is difficult to give accurate class predictions directly, we consider negative learning [Kim et al., 2019; Duan et al., 2022] to excavate knowledge from complementary labels which indicate the classes the image does not belong to. Concretely, we take an iterative manner to progressively store all the class predictions with the lowest probability, i.e., complementary labels, in a memory bank. Meanwhile, the model is also trained with the complementary labels during iteration, so as to exclude those impossible classes for the unreliable data. In addition, we design a negative consistency loss (NC-Loss) to constrain the complementary labels of different augmented views of the same unlabeled data to be consistent, intending to reduce the variations of complementary labels. By the above manner, all the unlabeled data can be fully used. Overall, our contributions can be summarized as follows：

• We propose a novel SS-DFER method LION to reduce the impact of ambiguous labels by exploring the potentially real labels in a candidate label set. To the best of our knowledge, this is the first work which considers the label ambiguity problem in SS-DFER.
• A progressive negative learning module is also presented in LION to draw the knowledge from the complementary labels, thus fully leveraging the underestimated unlabeled data with low-confidence. In addition, we innovatively design a NC-Loss to further improve the performance of the model.
• Extensive experiments on three challenging datasets show the effectiveness of our proposed model. Particularly, it sets new records of recognition accuracy with 67.83% on RAF-DB and 45.61% on SFEW, which is much higher than the second-best SOTA method [Li et al., 2022] by 5.47% and 3.73%.

2 Related Work

2.1 Facial Expression Recognition

FER helps computers to understand the emotional state of humans, which is meaningful for intelligent human-computer interaction. Previous works mainly experimented on the laboratory-generated dataset, including CK+ [Patrick et al., 2010], Oulu-Casia [Zhao et al., 2011], and achieved inspiring recognition accuracy. However, in the wild, it is difficult to guarantee that all images are of high quality. Therefore, more and more works [Wang et al., 2020; She et al., 2021; Xue et al., 2021; Yang et al., 2023] have extended the research interest to the in-the-wild datasets.

A troublesome problem in the in-the-wild datasets is the ambiguous images, that is, the given label does not match with the facial expression due to various factors. To address this problem, [Zeng et al., 2018] introduced multiple training stages to solve inconsistency of annotations. [Chen et al., 2020] explored label distribution through constructing auxiliary label space graphs. [Wang et al., 2020] focused on mining confidence weight and the latent truth of each sample for smaller impact of ambiguous data. Leading performance has been achieved by [She et al., 2021] which tried to find latent distribution in the label space and estimate the pairwise uncertainty.

Despite the achievements in the in-the-wild FER task, most of the current methods are fully supervised and require a large scale of well-labeled dataset, which is time-cost and laborintensive. [Florea et al., 2020] made the first attempt to investigate semi-supervised deep FER (SS-DFER) and proposed an extension of MixMatch [Berthelot et al., 2019]. [Li et al., 2022] proposed an adaptive thresholding approach to generate reliable pseudo-labels for high-confidence unlabeled samples. However, the SS-DFER is still mostly an unexplored research field. This work aims to tackle two main challenges in SS-DFER, including label ambiguity and inefficient utilization of unlabeled data with low-confidence.

2.2 Semi-Supervised Learning

Semi-supervised learning (SSL) appropriately frees up the stringent requirement of fully supervised learning for completely labeled dataset, while at the same time guaranteeing the performance of the model. Existing SSL methods can be divided into the following five main approaches: graph-based methods [Marino et al., 2016; Wang et al., 2020], generative model-based methods [Donahue et al., 2016; Denton et al., 2016], methods using consistent regularization [Sajjadi et al., 2016; Xie et al., 2020], methods using pseudo labeling [Rizve et al., 2021; Pham et al., 2021] and hybrid methods [Sohn et al., 2020; Zhang et al., 2021; Xu et al., 2021].

Among them, the hybrid methods generally achieve stateof-the-art performance on datasets such as CIFAR-10, CIFAR-100 [Krizhevsky et al., 2009], and ImageNet [Deng et al., 2009]. For example, [Xie et al., 2020] and [Sohn et al., 2020] set fixed thresholds to obtain pseudo labels for weakly augmented unlabeled images, and used them to supervise the prediction of the strongly augmented counterpart. [Zhang et al., 2021] and [Xu et al., 2021] further explored the application of dynamic thresholding in semi-supervised tasks. However, direct application of these SSL approaches to the SSDFER task is unsatisfactory due to the interference of ambiguous data and the underutilization of unlabeled data. In our work, we solve both the problems by proposing a disambiguation module and a progressive negative learning module.

3 Method

Our proposed LION aims to tackle the FER task in a semisupervised setting. Specifically, a label disambiguation module is devised to correct the ambiguous labels in the labeled set, while a progressive negative learning module is designed to make full use of unlabeled data, especially for those lowconfidence data. In this section, we first illustrate our problem formulation and show the overview of our method in Sec. 3.1. Then, the label disambiguation module of our method is introduced in Sec. 3.2. Furthermore, we describe how the progressive negative learning module works in Sec. 3.3. Finally, we display the whole training objective in Sec. 3.4.

3.1 Overview

In the semi-supervised setting, we are provided with a labeled 1=𝑖}𝑖ݕ ,𝑖ݔ{ = ௅ܦ set 1+ெ=𝑖}𝑖ݔ{ = 𝑈ܦ ெ and an unlabeled set ெ+ே , label one-hot corresponding is 𝑖ݕ image, input the is 𝑖ݔ where ,𝑈ܦ and ௅ܦ in data of number the are ܰ and ܯ classes, ܥ with as- correctly are }𝑖ݕ{ respectively. Notably, not all the labels a train to is purpose Our .}𝑖ݔ{ signed for their corresponding robust model to relabel the ambiguous labels and distinguish facial expressions accurately by fully leveraging both a large number of unlabeled images and few labeled images. Fig.2 gives an overview of our model. For a labeled image, we firstly apply weak augmentation (WA) to it and send it to a feature extractor to obtain the feature vector 𝑞, followed by a softmax function to derive its prediction probability . To remedy the ambiguous labels, the label disambiguation module first screens out the clear and ambiguous images separately according to the maximum probability in , i.e., . The clear images are used to compute the prototypes of all the expressions, guiding the relabeling process. All the ambiguous images will discard the original labels and be assigned with a candidate label set with multiple candidate classes, then approach the latent truth via the candidate labels and the prototypes. For an unlabeled image, both WA and strong augmentation (SA) are applied, and the weight-sharing feature extractor is employed to obtain the feature vectors 𝑞 , 𝑞 , and their prediction probabilities and , respectively. Then, based on the maximum probability in , i.e., , we divide the unlabeled images into two subsets. Specifically, if is greater than a certain threshold, pre- ̃ݕ we define it as a reliable sample and the pseudo label dicted from WA version will be used to supervise the SA version via the cross-entropy loss. Otherwise, we define it as an unreliable sample, and our proposed progressive negative learning strategy will be used to unearth the complementary labels which lead the learning of unreliable samples. Meanwhile, a negative consistency loss (NC-Loss) is proposed to enforce the consistency between the complementary labels of WA and SA versions, ensuring the accuracy and robustness of complementary labels. We will elaborate on key technologies in the following sub-sections.

3.2 Label Disambiguation Module

For a labeled image, we first generate a weakly-augmented version and obtain its prediction probability via the feature extractor and softmax function. If the maximum predicted probability exceeds a certain threshold and the class of define we ,ݕ is the same with the given label it as a clear image. Otherwise, it will be defined as an ambiguous one:

supervise to ݕ For the clear images, we use the given label the predicted probability :

∙ ܧܥ and images clear of number the represents 𝑙 ܰ where denotes the standard cross-entropy. In addition, since clear images have high-confidence labels, we also employ their feature vectors 𝑞 to calculate prototypes { } corresponding viewed be can prototypes These . ܥ …, ͳ, ∈ ܿ to each class as anchors of each class, which can guide the subsequent relabeling process. Particularly, the prototypes are updated in an Exponential Moving Average (EMA) fashion:

= ܿ = No lize(𝜙 + ͳ -𝜙 𝑞), if , (3) where is the prototype of last iteration, 𝜙 is set to 0.99 following [Wang et al., 2022].

As for each ambiguous image, we assign a candidate label one- the from Different it. to vector) C-dimension a (i.e., ݕ set set- classes multiple with vector multi-hot a is ݕ ,ݕ hot label ting to 1, any of which could potentially be the unique real follows: as ݕ label. We derive the candidate label set

, ݕ in class ht-ܿ the is ݕ where is the probability score corall manner, this In threshold. a is ߤ ,ܿ responding to the class possible candidate classes that could become the real label are selected to construct the candidate label set. To further approximate the real label, we make the label of each candidate class soft by assigning a weight propose and , ݕ to a moving-average style strategy to update the weight:

,ܿ where is initialized by the predicted probability of class similarity cosine the is ∙ ݀ is the weight of last iteration, between the feature representation 𝑞 and the prototype corresponding to the class . When the weight of one class in the candidate label set exceeds the threshold , the image will be relabeled as the most convinced class.

= ′ݕ if , ݕ . ൒ ݕ (6) Then it can be viewed as a clear image and optimize the model like Eq. 2.

3.3 Progressive Negative Learning

To fully leverage all unlabeled samples, we design a progressive negative learning module to explore the knowledge contained in them, especially for those low-confidence unlabeled samples, so as to further improve the performance of the model.

For each unlabeled sample, we transform it into both WA and SA versions. Then, we classify the unlabeled sample into reliable set or unreliable set according to its maximum probability of WA version, i.e., . If is larger than a threshold γ, the unlabeled sample is of high confidence for the model, thus belonging to the reliable set. Otherwise, the unlabeled sample with low confidence belongs to the unreliable set [Li et al., 2022]. We formulate this as follows:

The reliable samples have two applications. First, since model trusts the predictions of them with high confidence, we also utilize their feature vectors {𝑞 } to update the class prototypes by replacing 𝑞 in Eq. 3 with 𝑞 , thereby generating more robust prototypes. Second, all the reliable samples will be used to train the model like FixMatch [Sohn et al., 2020]. Concretely, we pick up the class with the maximum probabil= ̃ݕ ity as the pseudo label , then harness the pseudo label to supervise the prediction of the SA version, as follows:

samples. unlabeled reliable of number the denotes 𝑒𝑟ܰ where

As for the unreliable samples, although it is difficult to predict which expression category they belong to, there should be some categories with a sufficiently low probability score. Hence, it is easy for the model to know the classes these unreliable samples do not belong to, which are called complementary labels. To get as comprehensive a complementary label as possible, we maintain a complementary label bank (CLB) and update it progressively, as shown in Fig. 3. Specifically, given the prediction of an unreliable sample, we find its minimum probability score in . If in is smaller than a threshold 𝛿, we add its corresponding class ̅ݕ into the CLB, and accordingly get the complementary label as follows:

where  is a positive constant to ensure that there is sufficiently strong confidence to assign 1s in the complementary train to used is ̅ݕ label. Afterwards, the complementary label the model via negative learning as follows:

in ܿ where represents the predicted probability of class .

The above process will be performed iteratively until the remaining probabilities are all higher than the certain thresh-

Algorithm 1 The training process of LION

1=𝑖}𝑖ݕ , 𝑖ݔ{ = ௅ܦ Input: Labeled set = 𝑈ܦ ெ , unlabeled set }𝑖=ெ+1 𝑖ݔ{ ெ+ே .

thresh- the ,ܧ Parameters: The maximum number of epochs 𝛿. and , ,ߤ olds ,

Output: Updated LION model.

/* Training*/

• do ܧ ͳ: = ݁ 1:for

/*For labeled set*/

• do ܯͳ: 2: for 𝑖 =
• 𝑞 vector feature its obtain and 𝑖ݔ 3: Weakly augment .𝑖ݔ and prediction probability of
• 4: if ൒ nd then ݕ =
• 5: 2. Eq. by ௌܮ Compute
• 6: Compute prototypes by Eq. 3.
• 7: else
• 8: 4. Eq. by ݕ Assign a candidate label set
• 9: Soften the candidate labels by weights and update them by Eq. 5.
• 10: 6. Eq. by ′ݕ with 𝑖ݔ Relabel
• 11: end if
• 12: end for

/*For unlabeled set*/

• do ܰ+ܯ ͳ: +ܯ 13: for 𝑖 =
• fea- its obtain and 𝑖ݔ 14: Weakly and strongly augment ture vectors 𝑞 and 𝑞 as well as its prediction probabilities and .
• 15: if ൒ then
• 16: Update prototypes by Eq. 3.
• 17: 8. Eq. by 𝑒ோܮ Compute
• 18: else
• 19: while in ൑ 𝛿 do
• 20: Eq.9. by ̅ݕ Obtain the complementary label

21:

• 22: Re-compute 𝑞 , 𝑞 , , .

10. Eq. by ௅ேܮ Compute

• 23: end while
• 24: 11. Eq. by 𝐶ேܮ Compute
• 25: end if
• 26: end for
• 27:end for

old 𝛿. It is worth noting that the classes in previous complementary labels will not involve the probability prediction in the next iterations.

In addition, to further reduce the variations of complementary labels and enhance the robustness of the model, we propose a negative consistency loss (NC-Loss) between the WA ̅ݕ -∑ = 𝐶ேܮ and SA versions: ∈𝐶 lo ͳ- , (11)

from obtained label complementary the to refers ̅ݕ where the WA version and is the prediction of the SA version. For better understanding, we summarize the whole training process in Algrithm 1.

3.4 Overall Objective Function

As mentioned above, there are four losses to optimize the padata； clear labeled on loss ௌܮ rameters of our LION model: 1) WA the on loss ௅ேܮ 3) data; reliable unlabeled on loss 𝑒ோܮ 2) the between loss 𝐶ேܮ version of unlabeled unreliable data; 4) WA and SA versions of unlabeled unreliable data. Our LION model is optimized in an end-to-end process. To sum up, the total loss is formulated as follows:

,𝐶ேܮ ߣ+ ௅ேܮ ߣ + 𝑒ோܮ1ߣ + ௌܮ = 𝑙𝑎𝑡𝑜𝑡ܮ (12) these balance to hyper-parameters are ߣ and , ߣ ,1ߣ where terms.

4 Experiments

4.1 Datasets and Metrics

Datasets. We verify the effectiveness of our LION model on three datasets: RAF-DB [Li et al., 2017], AffectNet [Mollahosseini et al., 2017], and SFEW [Dhall et al., 2011]. RAFDB is constructed by 30,000 facial images which are annotated by 40 experts. In the experiment, we select fear, surprise, sadness, happiness, disgust, anger and neutral as classification categories. The sizes of the training and test sets are 12,271 and 3,068, respectively. AffectNet has 420,000 face images containing 8 manually annotated expression labels. We choose the same seven categories as RAF-DB in experiment and use 240,000 images for training and 3,500 images for validation. SFEW contains static frames extracted from movies including 958 training images and 436 test images. To simulate the semi-supervised setting, we randomly discard a portion of labels at different ratios.

Performance Metrics. We conduct experiments using different random seeds and calculate the mean accuracy and standard deviation on the test set to evaluate the performance of the method.

4.2 Implementation Details

By default, ResNet-18 is used as backbone network, which is pre-trained on MS-Celeb-1M face recognition dataset [Guo et al., 2016]. Facial images are aligned and resized to 224×224 by MTCNN [Zhang et al., 2016]. RandomCrop and RandomHorizontalFlip are employed as weak augmentation. RandAugment [Cubuk et al., 2020] is used as strong augmentation following [Li et al., 2022]. The whole network is trained for 20 epochs with the Adam optimizer. The initial learning rate is set to 5 × ͳͲ- . The batch size is 16. The above setting keeps consistent with that of all the compared methods for fairness. As for the hyperparameters, we set parameters trade-off the and 3.0=ߤ𝛿=0.05, =0.80, =0.83, respectively. 0.06, 0.45, 0.85, to set are ߣ , ߣ ,1ߣ

4.3 Comparison With the State-of-the-Art

To test the performance of LION, we compare it with several state-of-the-art methods, including Pseudo-Labeling [Lee et al., 2013], MixMatch [Berthelot et al., 2019], UDA [Xie et al., 2020], Margin-Mix [Florea et al., 2020], ReMixMatch [Berthelot et al.,2020], FixMatch [Sohn et al., 2020], and Ada-CM [Li et al., 2022], on all the three datasets with different ratios of labeled data. All these methods are highly representative and influential in the field of semi-supervised image classification, and we have tailored them to adapt our semi-supervised deep FER (SS-DFER) task. Particularly, we regard our model trained using only limited labeled data as the baseline. And we apply DLP-CNN [Li et al., 2019] on RAF-DB and SFEW, RAN [Wang et al., 2020] on AffectNet as fully-supervised baseline.

Table 1 shows the comparison results. From the table, it is clear that all the semi-supervised methods achieve better performance than baseline due to the utilization of unlabeled data. Compared with the second-best method Ada-CM, our method outperforms it overwhelmingly under all ratios of labeled data on all datasets. Even when only 100 images are labeled in RAF-DB and SFEW, our method still leads the performance by 5.47% and 3.73% accuracy, showing the powerful capability of our method to make use of unlabeled data. Furthermore, compared with the fully supervised result, our method can still beat the baseline with a large margin, i.e., 0.68% on RAF-DB, 3.13% on SFEW, 6.14% on AffectNet when 4000, 400, 10000 labeled samples are used. The above results show that our model makes fuller use of the unlabeled data and reduces the negative impact caused by the ambiguous data.

4.4 Ablation Study

In this section, we carry out ablation study to verify the contribution of components and investigate the optimal values of hyper-parameters in LION.

Effectiveness of Components in LION. There are two important components in our LION model: 1) the label disambiguation module (LD for short) and 2) the progressive negative learning module. We intend to ablate them from the whole module to evaluate their contribution. Particularly, we attribute the contribution of the progressive negative learning base- a construct We loss. 𝐶ேܮ the and loss ௅ேܮ module to the 𝐶ேܮ the and loss ௅ேܮ line by removing the LD module, the loss. The experiment is performed on RAF-DB and SFEW with 100 labels and 400 labels, respectively. From Table 2, we can observe that the LD module significantly improves the performance (e.g., +2.98% on RAF-DB and +3.93% on SFEW), which signifies its capability in reducing the impact of ambiguous labels and optimizing the decision boundary. performance the incorporated, is 𝐶ேܮ or ௅ேܮ Moreover, when further rises by 3.87% on RAF-DB or 0.79% on SFEW. Both losses are employed simultaneously together with the LD module will bring greater performance gains. This is because both the modules are actually conducive to each other. Specifically, the progressive negative learning module enables the model to learn more knowledge from unlabeled data, which can encourage the model to give more reliable predictions. These reliable predictions provide more accurate information to the prototypes, helping the LD module correct more ambiguous labels. Correspondingly, more clear data will also optimize the decision boundary of the model, thus promoting the progressive negative learning module to work more effectively. In summary, all these results fully demonstrate the effectiveness of the proposed components.

Evaluation of the Parameter 𝝉 . The parameter is the threshold to determine if the data is ambiguous or not. We investigate its effect under values in [0.6,0.9]. Figure 4(a) shows that the performance is positively correlated with the increasing from 0.6 to 0.8. When exceeds 0.8, the performance degrades. This is because a large or small will guide the model with either wrong ambiguous data or clear data. Accordingly, we set as 0.8.

Evaluation of the Parameter 𝜸. The parameter is used to classify the unlabeled data as reliable or unreliable ones. We investigate its effect under values in [0.77,0.89]. From Figure 4(b), we can see that the best result is achieved when =0.83. When is too small, more data with wrong pseudo labels are considered reliable and the updates of prototypes will be affected.

4.5 Visualization

In this section, we show some ambiguous examples relabeled by our method. In Figure 5, the blue represents original ambiguous label and the red represents corresponding corrected label by our LION model. We can observe that the corrected labels by our model are more in tune with human intuition. For example, the girl in the third column obviously has signals like a tight face and a closed mouth, which are barely possible to be "surprise" from the perspective of human. On the contrary, the corrected label "fear" is more suitable for these signals.

5 Conclusion

In this paper, we propose a novel SS-DFER method LION which includes a label disambiguation module and a progressive negative learning module. The label disambiguation module corrects the given ambiguous labels based on the candidate label set and prototypes. Progressive negative learning module mines complementary labels more completely during iteration. In addition, a negative consistency loss (NC-Loss) is proposed for a more robust model by reducing the variations of complementary labels. Extensive experiments conducted on three challenging datasets show that LION achieves state-of-the-art results and surpasses fully-supervised baselines.

Acknowledgments

This work is supported by National Natural Science Foundation of China (NSFC 62071314), Sichuan Science and Technology Program 2023YFG0263, 2023NSFSC0497, and Opening Foundation of Agile and Intelligent Computing Key Laboratory of Sichuan Province.

Contribution Statement

Zhongjing Du and Xu Jiang contributed equally to this work. Yan Wang is the corresponding author to the work.