Cross-Domain Facial Expression Recognition via Disentangling Identity Representation

1 Introduction

Due to the difference in photographic environments (e.g., inthe-lab or in-the-wild) and collected subjects, there are obvious domain shifts across different facial expression datasets [Li and Deng, 2020]. The current well-performing FER methods may achieve satisfactory performance in intradataset protocols, but their performance drops dramatically in inter-dataset settings [Recht et al., 2019; Zhang et al., 2021]. Recently, a series of cross-domain FER algorithms [Chen et al., 2021; Li et al., 2022; Li et al., 2021] have been widely developed to address this problem. They mainly focus on exploiting the idea of domain adaptation (DA) to collect data from each possible target domain and train the model with each source-target pair [Kang et al., 2019; Huang et al., 2021b]. However, in practice, there may not be any target domain data available during the training of expression recognition for the model to analyze the distribution shift to overcome the negative effects.

Recently, the DG methods have been studied extensively, which aim to generalize the knowledge extracted from multiple source domains to an unseen target domain that is not accessible during training [Gan et al., 2016; Tzeng et al., 2017]. Existing DG methods attempt to learn domain-agnostic features via employing various strategies such as adversarial feature learning [Li et al., 2018], domain adversarial image generation [Zhou et al., 2020], domain randomization [Huang et al., 2021a], or style mixing [Zhou et al., 2021b]. However, the domain-invariant features extracted by these methods may contain identity information, which makes the discrimination of the learned expression features limited. Eventually, the generalization of cross-domain FER is weakened. In fact, the identity information here is often the focus of face recognition or face verification work. In addition, the expressions to be recognized in real-world applications often come from unknown domains and subjects. It is necessary to eliminate the reliance on domain and identity information in the expression recognition process. As shown in Figure 1, cross-domain FER is interfered by the domain (different datasets) and identity information. Existing DG methods can hardly remove the interference of identity information when learning domaininvariant features. Therefore, the discriminative capability of their extracted expression features is limited.

To solve the above issue, we extract more discriminative expression features via disentangling domain, expression, and identity information from the aspect of feature disentanglement [Zhou et al., 2021a]. Thus the generalization performance of cross-domain FER is improved. Related studies [Muandet et al., 2013; Zhang et al., 2022] show that the learned representations reflect the intrinsic category semantics and have strong invariance to cross-domain variation, which is of great importance to the robustness and generalization of deep learning. Li et al. [Li et al., 2022] also demonstrate that the learned representation is separable and domain-invariant, which is effective for cross-domain FER. Therefore, we handle cross-domain FER via learning domain-invariant, separable, and discriminative representations. As shown in Figure 1, we propose a representation disentanglement framework for extracting domain, expression, and identity features, respectively. Disentangling identity features from domain and expression features can enable our model to extract discriminative expression features in unseen domains. Moreover, to enforce the disentanglement of the above information and reduce the coupling between features, we provide additional supervision, which renders the disentangled expression features to be invariant to interference factors. Eventually, cross-domain FER can be better performed.

Additionally, we know from visual psychophysics that changes in the amplitude of the image Fourier transform can significantly alter the appearance but not the interpretation of the image. The high-level semantic information in images is associated with the phase of its Fourier transform [Hansen and Hess, 2007; Yang et al., 2020]. Inspired by this, to ensure that the high-level semantic information of the facial image remains unchanged after disentangling domain information, we recover the encoded expression and identity features as images reconstructed from Fourier phase information. Accordingly, we construct the generator with an encoder-decoder structure that acts as an image semantic changer. It aims to output the fake Fourier phase information reconstructed image that can fool the phase information discriminator. Besides, to smooth the transformation of expressions, we introduce the identity discriminator to control the distribution of identity features. With the additional expression classifier, the decoder can strive for the generated image to have the same semantics as the real Fourier phase information reconstructed image. The above operations facilitate the disentangling of identity features and improve the discriminative ability of expression features. Meanwhile, they can ensure that the extracted expression features have good domain invariance.

The contributions of our work can be summarized as follows. (1) We propose a representation disentanglement network for domain-generalized FER, which can recognize facial expressions in unseen domains during inference. (2) We disentangle the identity features from both domain and expression features. Furthermore, the identity and expression features are recovered as the Fourier phase information reconstructed images. As a result, the discrimination and domaininvariance of expression features are all enhanced, which improves the performance of cross-domain FER. (3) With welldesigned supervised learning strategies, the network modules are able to joint learning efficiently. Extensive experiments are conducted on different datasets. And the analytical results demonstrate the effectiveness and superiority of our method.

2 RelatedWork

2.1 Cross-Domain Facial Expression Recognition

To address the prevalent domain discrepancies among different FER datasets, some cross-domain FER algorithms have been recently proposed. For instance, Zhu et al. [Zhu et al., 2016] propose a discriminative feature adaptive method to learn a feature space to represent facial images from different domains. Li et al. [Li and Deng, 2018] present a Deep Emotransfer Network to reduce the bias among the datasets. Chen et al. [Chen et al., 2021] integrate graph propagation with adversarial learning mechanisms to learn domain-invariant holistic-local features for cross-domain FER. Li et al. [Li and Deng, 2020] propose a deep Emotion-Conditional Adaption Network to learn domain-invariant representations. However, the above methods require depending on the target domain data (without labels) to analyze the domain distribution shift. Li et al. [Li et al., 2022] present a Deep Margin-Sensitive representation learning framework to extract multi-level features. But it depends highly on the supervision information on the target domain data since it needs to generate accurate pseudo-target labels. DLP-CNN [Li et al., 2017b] aims to enhance feature discrimination by maximizing the interclass dispersion while preserving local closeness. Zavarez et al. [Zavarez et al., 2017] analyze the performance impact of fine-tuning with cross-database methods. Ji et al. [Ji et al., 2019] propose an intra- and inter-class feature fusion network for FER across datasets. However, these approaches do not consider the influence of identity information on expression representation, thus limiting the discriminative power of expression features.

Different from the above approaches, we focus on the cross-domain FER when the target domain data is inaccessible, which is a more difficult case. We eliminate the dependence on identity during domain-invariant expression feature learning to improve the performance of cross-domain FER.

2.2 Domain Generalization

To overcome the problems of domain shift and lack of target domain data, DG is introduced. It aims to use the data from single or multiple related but different source domains to train the model that can generalize well to unseen target domains. It is evident that the DG model must depend only on the source domain to learn the domain-invariant representation. DG methods have been applied to many tasks, such as object recognition [Li et al., 2018], image segmentation [Huang et al., 2021a], and face anti-spoofing [Wang et al., 2022]. Researchers categorize existing DG methods into different groups based on design motivations [Zhou et al., 2021a], e.g., Domain Alignment [Shao et al., 2019], Ensemble Learning [He et al., 2016], Learning Disentangled Representations, etc. Here we only briefly review the DG methods relevant to Learning Disentangled Representations. For example, Chen et al. [Chen et al., 2016] utilize information-maximizing generative adversarial networks to learn disentangled representations. Chattopadhyay et al. [Chattopadhyay et al., 2020] suggest learning domain-specific masks and encourage masks to learn a balance of domain-invariant and domain-specific features. Piratla et al. [Piratla et al., 2020] propose a CSD that jointly learns public and domain-specific components. Since the facial expression is closely entangled with identity information, it is difficult for the above DG methods to disentangle them when learning the domain-invariant representation. Identity and expression representations are not disentangled, which limits the discriminative ability and generalization performance of expression features.

In contrast to the above DG methods, we analyze and consider the effect of identity information for the cross-domain FER task. Domain labels and generated Fourier phase information images are leveraged to disentangle features, which facilitates the model to learn domain-invariant, separable, and discriminative expression features.

3 Methodology

Forcing the entire model or features to be domain-invariant during cross-domain FER is challenging. We relax this constraint by allowing some parts to be domain-specific, i.e., learning disentangled representation. Intuitively, we learn three independent potential subspaces of the domain, expression, and identity information from facial images. In addition, the adversarial way is utilized to assist the model in learning domain-invariant and discriminative expression features.

3.1 Overview

Given a training set Ds = {D1,D2, · · · ,DS} from multiple source domains, our goal is to learn a domain-agnostic FER model that is expected to perform well on the disjoint and unseen target domains Dt. There are Nj labeled facial expression images {(xji , yj i )}Nj i=1 in the j-th source domain Dj, where xji and yj i ∈ {0, 1, · · · ,K} (K = 6 means the number of expression categories is 7) represent the input images and expression labels, respectively. The facial expressions in Dt will be recognized during the inference stage.

As shown in Figure 2, there are three encoders in our proposed network architecture for processing the input facial image: domain encoder Ed, expression encoder Ee, and identity encoder Eid. The domain encoder Ed is designed to extract the domain representation fd from the input image, and the classifier Cd performs the domain classification task based on this representation. The expression representation fe is extracted by the expression encoder Ee, followed by the expression classifier Ce for performing FER. The identity encoder Eid extracts the facial identity representation fid from the images, while the decoder De is used to reconstruct the guarantee. The image still retains high-level semantics after disentangling the domain information, which can ensure further disentanglement of expression and identity representations. Inspired by the semantic preservation property of the Fourier phase component [Oppenheim and Lim, 1981;

Xu et al., 2021], the phase-only reconstructed images ( ˆX) and phase information discriminator Dp are taken to guarantee the high-level semantics invariance of the images (Xp) reconstructed by the decoder De. The additional identity discriminator Did controls the distribution of identity features and facilitates the smooth transformation of expressions. Once the joint learning of the above modules is completed, the crossdomain FER task can be accomplished simply through inference with Ee and Ce.

3.2 Learning Image Domain Representation

To address the domain-generalized FER, we suggest learning image domain representation from expression images. As shown in Figure 2, the learning of the domain representation is implemented by the Ed and Cd. Where the Ed is expected to describe information about background, illumination, resolution, etc. And the Cd recognizes the image domain j (j ∈ {1, 2, · · · , S}) based on such features fd (fd = Ed(x)). It indicates which source domain the input image comes from. We define the domain classification loss Ldom as follows.

where mj denotes the ground truth of the image domain label. Moreover, we provide additional supervision to ensure that the domain representation learned by Ed and Cd do not contain expression information. Specifically, with the deployment of the expression classifier Ce, we let Ce take the features fd. And the Ce is not expected to perform expression recognition on the domain features. Therefore, we provide the following auxiliary confusion loss Lconf dom .

With the above Ldom and Lconf dom , our proposed framework can disentangle the domain features from the input expression images. In addition, the deployment of Ed and Cd also facilitates expression and identity learning.

3.3 Learning Expression and Identity Representations

Since identity information is useless for cross-domain FER, we propose to decouple it, which improves the discrimination of expression features. Extracting expression and identity representations separately from the input image is the main component of our approach. With the identity and domain representations correctly disentangled from the input image, our learned expression features can be effectively applied to recognize facial expressions in unseen target domains.

For the learning of expression features, as shown in Figure 2, the expression encoder Ee and classifier Ce are deployed in our framework to achieve this goal. Ee is expected to extract expression features (fe = Ee(x)). Ce classifies expressions of the input image based on the features fe. To better separate each expression, we utilize the simplified LMCL function

[Wang et al., 2018] as the objective function, i.e., the expression classification loss Lexp is defined as follows.

where α is the hyperparameter. W = {Wk | k = 0, 1, · · · ,K} represents the parameters of the Ce. The separation between the features of each expression can be further enforced by introducing the margin m.

Similarly, to further ensure that the expression features would not contain any image domain information, we provide additional supervision - the expression confusion loss Lconf exp , defined as follows:

For the learning of identity, the identity encoder Eid is designed to learn the mapping of facial image to identity representation (fid = Eid(x)). The high-level semantics of the image needs to be preserved after disentangling the domain representation, which facilitates the disentanglement of identity and expression information. We leverage the semantic preservation property of the Fourier phase component to provide support for preserving the high-level semantics. Note that using only Fourier phase information for cross-domain FER is not satisfactory because it still suffers from the interference of identity information. As shown in Figure 3, the facial images from different domains are processed in the following two ways. One way is the reconstruction of image with amplitude information by setting the Fourier phase component to a constant. The other is the reconstruction of image with phase information by setting the Fourier amplitude component to a constant. It can be seen that the Fourier phase information reconstructed image retains the semantics of the original image.

For example, the Fourier transformation F(x) of a single channel image x is formulated as:

The amplitude and phase components are denoted as:

where R(x) and I(x) denote the real and imaginary part of F(x), respectively. Set amplitude to be a constant. The constant we use here is the standard deviation of amplitude.

The constant amplitude spectrum is combined with the original phase spectrum to form a new Fourier representation, as shown in the following formula:

It is then sent to the F -1 (x) to generate the image, i.e., the reconstructed image with Fourier phase information is represented as follows. F -1 (x) is the inverse Fourier transformation that maps the spectral signal back to the image space. F(x) and F -1 (x) can be calculated with the FFT algorithm [Nussbaumer, 1981] efficiently.

Then, we adopt the idea of generative adversarial and take the identity features fid and expression features fe as the input to the decoder De which is designed to recover (generate) the images reconstructed with Fourier phase information. The discriminator Dp is designed to distinguish the fake images generated by De from the real Fourier phase information reconstructed images. Besides, De and Dp assist the expression encoder Ee and the identity encoder Eid in learning the disentanglement representation from facial images, i.e., their deployment contributes to the disentanglement of fid and fe. The min-max objective function is defined as follows:

Furthermore, we provide phase information similarity loss to ensure that the output image of decoder De shares expression and identity representations with the input phase information reconstructed image.

The identity discriminator Did imposes a uniform distribution on the identity features fid, which contributes to smooth the expression transformation. Assume that Prior(f) is the prior distribution and f ∗ id ∼ Prior(f) represents the random sampling process from Prior(f). The min-max objective function adopted is presented in the following formula:

Similar to the design of domain and expression features, we need to ensure that the identity representation learned by the Eid does not contain domain and expression information. With the deployment of Cd and Ce, we propose the following identity confusion loss Lconf id :

Together with the above classification loss, adversarial loss, similarity loss, and confusion loss, we train our proposed framework to be able to disentangle identity and expression representations. Finally, the expression representation we learned is domain invariant and discriminative owing to the disentanglement of the image domain and identity information. Therefore, our model can be effectively generalized to FER across different domains.

In summary, our overall objective function is defined as follows:

where TV (·) represents the total variation that effectively eliminates ghost artifacts. The coefficients λ and τ balance smoothness and resolution. The network is trained and updated by the above formula. Once the training of our network architecture is complete, only the expression encoder Ee and classifier Ce will be utilized to perform cross-domain FER. That is, Ee is applied to extract domain-invariant and discriminative expression features. Then those features are fed into Ce for facial expression prediction.

4 Experiments

4.1 Datasets

JAFFE [Lyons et al., 1998]: The JAFFE (denoted as J) is a laboratory-controlled facial expression database that contains 213 samples from 10 Japanese females. Each person has 3-4 images that are annotated with seven basic facial expressions (anger/AN, disgust/DI, fear/FE, happiness/HA, neutral/NE, sadness/SA and surprise/SU). We follow previous works [Chen et al., 2021] to use the entire dataset for the training and test sets.

Oulu-CASIA [Zhao et al., 2011]: The laboratorycontrolled Oulu-CASIA (denoted as O) is a database of 2,880 image sequences from 80 subjects captured with two imaging systems under three different illumination conditions. These subjects are labeled with seven basic facial expressions. As in [Li et al., 2022], we use 1,920 images from Oulu-CASIA VIS which is obtained by selecting the last three frames and the first frame (neutral expression) from 480 videos with the VIS system under normal indoor illumination.

RAF-DB [Li et al., 2017b]: The RAF-DB (denoted as R) is a real-world database consisting of around 30,000 facial images annotated with seven basic or eleven compound expressions. In the experiment, we select images labeled with seven basic expressions, 12,271 of which are used for training and 3,068 for testing. In addition, RAF-DB 2.0 [Li and Deng, 2020] (denoted as R2.0) is an extension of the current RAFDB database. A subset of R2.0 contains 14, 216 images that are utilized as source data in the experimental comparisons.

SFEW 2.0 [Dhall et al., 2011]: The in-the-wild database SFEW 2.0 (denoted as S) is the benchmark dataset for the SReco sub-challenge in EmotiW 2015 [Dhall et al., 2015]. Each image is assigned one of the seven basic expressions. We use 958 images for training and 436 images for testing.

4.2 Experimental Results

Leave-One-Domain-Out Results

Following [Li et al., 2017a], we first perform the leave-onedomain-out evaluation. Specifically, we train our model using labeled images from multiple source domains and choose the best model on the validation splits of the training set. For testing, we evaluate the selected model on a retained target domain. For example, "O&R&S to J" means that O, R, and S are treated as the source domains, and J is used as the target domain. As shown in Table 1, we provide the experimental results for different backbones (ResNet-18 and ResNet-50). We use a network consisting of the Ee and Ce as the baseline. And we compare our approach with some state-of-the-art DG methods that are not specifically designed for cross-domain FER. The superiority of our approach demonstrates that training model to disentangle expression and identity features can improve its performance on unseen domain images.

Comparison with Other Cross-Domain FER Methods

Following the setup of previous works [Chen et al., 2021; Li et al., 2022], we compare the proposed method with other cross-domain FER approaches, and the results are shown in Table 2. Since our model uses domain labels, the training set requires multi-source domain data. To be consistent with the size of training set in other methods, we select a subset of the multi-source domain (R and O) to be our training set. Also, the selected source and target domains are not overlapped. It can be seen that our method outperforms the existing methods. Other cross-domain FER methods still have limited cross-domain recognition performance due to their reliance on identity information. Our method achieves impressive results even though the target domain is not accessed during training. The comparison shows that our proposed model can learn expression features with strong discriminative ability by disentangling expression-irrelevant information. Therefore, the cross-domain FER performance is improved.

Ablation Study

To demonstrate the importance of each module in our proposed framework, we conduct the following ablation studies on different target domains. As shown in Table 3, we provide experimental results of models "a" (Baseline (Ee,Ce)), "b" (Baseline+Ed,Cd), "c" (Baseline+Eid,Did,Dp,De), and "f" (Ours), respectively (the backbone is ResNet-50). The comparison of models "b" and "a" confirms that the disentanglement of image domain features assists our model in extracting domain-invariant facial content. The comparison of models "c" and "a" verifies that the disentanglement of identity information helps our model focus on facial expression, thus improving the discriminative ability of expression features. Moreover, the effectiveness of leveraging Fourier phase information to help disentangle identity features is also demonstrated. The comparison of models "f" and "a" shows that our proposed representation disentanglement network can learn domain-invariant and discriminative expression features, thus bringing significant improvements. Models "d" (w/o confusion loss) and "e" (w/o Lsim p +TV (·)) are ablation studies of the loss function, illustrating the effectiveness of our introduction of confusion loss, similarity loss, and TV (·). In addition, the identity discriminator Did is introduced to smooth the transformation of expressions. As shown in Figure 4, we provide the experimental results before and after the introduction of the Did. The effectiveness of introducing the Did can be observed.

Visualization

We visualize the extracted expression features to reflect their domain invariance. Furthermore, to enforce the disentanglement among features, the auxiliary confusion loss is introduced. Therefore, we provide the t-SNE [Van der Maaten and Hinton, 2008] visualization results as in Figure 5. Here we choose R as the target domain and perform leave-onedomain-out experiments (J&O&S to R, backbone is ResNet50). Specifically, we train without and with confusion loss, respectively, and then obtain the trained Ee and Ce. During testing, the images in the target domain (we choose around 100 facial images per expression) is randomly selected as the input of the trained Ee to extract expression features. The acquired features are visualized separately in Figure 5. We can see that our approach can well capture the domain-invariant expression features. Also, the introduction of the confusion loss helps to improve the performance of cross-domain FER.

Our method extracts expression features by disentangling identity and domain information. To further verify its effectiveness, we use the Grad-CAM [Selvaraju et al., 2017] algorithm to obtain the class activation mapping visualizations. We select O and R as the target domain for the leaveone-domain-out experiments (J&R&S to O and J&O&S to R, backbone is ResNet-50). The class activation maps for each expression in different domains is shown in Figure 6. The image regions that the model focuses on during the crossdomain FER can be seen. The (a) and (b) denote models "b" and "f" (Ours) in Table 3. We find that model "b" can focus on the facial region without the influence of domain information. And model "f" can locate more expression-related potential interesting areas by disentangling the identity features. That is, our whole disentanglement model can focus on domaininvariant and more discriminative facial regions. Therefore, the performance of cross-domain FER is improved.

5 Conclusion

In this paper, we attempt to solve the more challenging task of cross-domain FER not being able to access the target domain data. We propose a representation disentanglement model capable of extracting domain, identity, and expression features based on the idea of DG. To ensure the disentanglement among features, we use the semantic preservation property of the Fourier phase component to provide support for preserving the high-level semantics of the image. In addition, we introduce classification loss, adversarial loss, similarity loss, and confusion loss, respectively. And the training is implemented by our designed objective loss function. Extensive experiments on multiple datasets demonstrate the effectiveness of our method.

Acknowledgements

This work is partially supported by National Natural Science Foundation of China (No.62206116), Key Research and Development Program of Hubei Province (2022BAA079), Major Science and Technology Projects of Jilin Province (20210301030GX), and Leading innovation and entrepreneurship team of Zhejiang (2019R01015).