Micro-Expression Recognition Enhanced by Macro-Expression from Spatial-Temporal Domain

1 Introduction

Micro-expression (ME) occurs when a person either deliberately or unconsciously conceals his or her genuine emotions [Ekman, 2009]. Compared to large intensity and long duration characteristics of macro-expression, micro-expressions is brief and subtle. Since it can be applied to many areas such as national security, clinical diagnosis and judicial system, automatic micro-expression recognition (MER) has become an active research area in recent years.

Micro-expression recognition can be classified into two categories: handcraft feature methods and deep feature methods. Researchers usually use Histogram of Oriented Gradient (HOG), Histogram of Optical Flow (HOOF) and Local Binary Pattern-Three Orthogonal Planes (LBP-TOP) to extract handcraft features. Li et al. [2017] used feature differences for ME spotting and adopted the histogram of image gradient orientation-TOP (HIGO-TOP) for MER. Liong et al. [2018] introduced a Bi-Weighted Oriented Optical Flow (Bi-WOOF) based feature extractor, while Happy et al. [2017] proposed a fuzzy HOOF method (FHOFO), which ignored the subtle motion magnitudes and only took the motion direction into consideration. Wang et al. [2014] adopted a pruned LBP descriptor using six neighbors around every point (LBP-SIP) which reduces the inherent redundancy within LBP-TOP. Huang et al. [2015] adopted an integral projection method to boost the capability of LBP-TOP (STLBP-IP) by supplementing shape information. Le et al. [2016] used LBP-TOP to learn significant temporal and spectral structures with sparsity constraints. Due to the short duration and low intensity of ME, handcraft features are not robust in the micro-expression identification and classification.

∗Contact Author

Recently researchers use deep network for MER as the micro-expression databases gradually developed. Peng et al. [2019] explored their underlying joint formulations and proposed a consolidated Eulerian framework to reveal the subtle facial movements. It expanded the temporal duration and amplified the muscle movements in ME simultaneously. Van Quang et al. [2019] used the newly proposed framework CapsuleNet [2017] to figure out the part-whole relationships for MER. Khor et al. [2019] proposed a lightweight dual-stream shallow network (DSSN) in the form of a pair of truncated CNNs with heterogeneous input features. Liong et al. [2019] designed a shallow triple stream threedimensional CNN (STSTNet) to extract details of ME, with optical strain, horizontal and vertical optical flow as input. Lei et al. [2020] applied learning-based video motion magnification to magnify ME and designed graph-temporal convolutional network (Graph-TCN) to extract the features of the local muscle movements. To fully exploit the dependence between action units (AUs) and expression, Sun et al. [2020] proposed a knowledge transfer technique that distilled and transferred multi-knowledge from AU for MER. Xie et al. [2020] proposed AU-assisted graph attention convolutional network (AU-GACN) for MER, which effectively integrated AU recognition. Nevertheless, these deep learning methods suffer from insufficient micro-expression training samples.

Nowadays there are many macro-expression databases, which contain a large number of labeled training samples.

Although macro-expression has longer duration and higher intensity than micro-expression, these two expressions share some similarities in spatial and temporal facial behavior patterns. Figure 1 shows a comparison between micro and macro expressions. As shown in the expression sequences varying from onset frame to apex frame, we can obviously find that both micro and macro expression gradually raise the lip corners for happiness. For surprise, both image sequences raise eyebrows and open eyes. Thus, how to transform macroexpression information for micro-expression recognition has become an important research direction.

Peng et al. [2018] pretrained a deep network frommacroexpression images, and fine-tuned it with micro-expression images by transfer learning protocols. However, they did not consider the gap between micro and macro expression images, which limits the effect of transfer learning. Liu et al. [2019] used Expression Magnification and Reduction (EMR) to reduce the gap of apex frame between micro and macro expression visually, and trained on a fusion of micro and macro expression database (Neural). Since EMR actually introduces a lot of noise to micro-expression, it can't guarantee the similarity of micro and macro expression features. Xia et al. [2020] introduced disentangle network to extract expression-related embedding from apex frame, and used loss equality regularization to transform macro-expression information (MicroNet). However, their method can't be trained end-to-end, which impairs the performance of overall framework. The above macro-expression assisted methods only use spatial features to capture static structure patterns, which ignore dynamic muscle movement from temporal features.

In order to tackle the aforementioned challenges, in this paper we propose a macro-to-micro transformation model which enables to transfer spatial and temporal pattern existed in macro-expression to micro-expression recognition. As shown in Figure 2, we first pretrain two-stream baseline model for micro-expression data and macro-expression data respectively, named MiNet and MaNet. In order to take advantage of macro-expression data, we model the shared features in the spatial and temporal domain simultaneously. In spatial domain, we introduce a domain discriminator to align the features of MiNet and MaNet, so MiNet can capture static textures of facial appearances better. In temporal domain, we introduce a relation classifier to predict the correct relation for temporal features of different sampling interval fromMaNet and MiNet. Through this task, MiNet can learn the dynamic pattern of muscles from MaNet. Finally, we explicitly take the class label into account and introduce contrastive loss to encourage the MiNet to pull the same class samples together in the feature space, while simultaneously pushing apart clusters of samples from different classes.

In summary, our contributions are two-folds: 1) We propose a well-designed macro-to-micro transformation framework by two auxiliary tasks from spatial and temporal domain respectively. 2) We utilize contrastive loss to learn a more robust clustering of the feature space for micro-expression and macro-expression.

2 Method

2.1 Problem Statement

Suppose we have micro-expression data DI = {x(j) i , y(j) i }Nj=1 , and macro-expression data DA = {x(j) a , y(j) a }Mj=1 . DI contains N training instances of micro-expression and DA contains M training instances of macro-expression. xi and xa respectively represent the micro and macro expression videos that change from neutral-face to emotional-face. yi, ya ∈ {1, 2, · · · ,L} are the expression label, L is the number of expression category. Our goal is to train a deep network for micro-expression recognition with the help of macro-expression data in both spatial and temporal domain. During testing, only micro-expression data is required.

2.2 Baseline Model

Videos can be decomposed into spatial and temporal components. The spatial part, in the form of individual frame appearance, carries static structure patterns about face depicted in the video. The temporal part, in the form of motion across the frames, conveys the movement of the facial muscle.

We use two-stream network as our baseline model, which contains spatial stream network, temporal stream network and expression classifier. We pick out the apex frame from the expression video and input it to the spatial stream network. The spatial stream network operates on individual image frame, effectively extracting apex-level feature f A from apex frame xA. The appearance from apex frame contains useful clue, since some facial expressions are strongly associated with particular facial muscle. The temporal stream network is proposed to capture dynamic patterns from consecutive frames. The temporal stream network first extracts frame-level features f = {f 1 , f 2 , · · · , f T } from expression video, and T is the length of video. Then we aggregate framelevel features with mean-pooling along the temporal direction to generate the video-level feature v. We fuse the apex-level feature f A and video-level feature v to get fused feature z, and input it into expression classifier Ce.

Two two-stream networks are built in our framework. The first one performs micro-expression recognition from microexpression data DI , and is the desired classifier in this paper, named MiNet. The second one performs macro-expression recognition from macro-expression data DA, named MaNet. These two networks are pretrained with the supervised multiclass cross entropy loss as shown in Eq.1:

1B ∈ {0, 1} is an indicator function that returns 1 if B evaluates as true. After pretraining, MiNet is further fine-tuned under the guidance of MaNet.

2.3 Guidance from Spatial Domain

The spatial stream network can process the static facial textures, which have a strong correlation with expression. Limited by small datasets of micro-expression, the spatial stream of MiNet may not capture static textures of facial appearances well. Thus, we propose an auxiliary task to learn effective feature from macro-expression in spatial domain.

Inspired by one of the most popular adversarial-based approaches DANN [Ganin and Lempitsky, 2015], we adopt adversarial method to align the spatial features of MiNet and MaNet. Specifically, for the spatial features from the MiNet and MaNet, we introduce spatial domain discriminator Ds to align macro-expression features f A a and microexpression features f A i . Through adversarial training with a gradient reversal layer (GRL) which reverses the gradient signs during back-propagation, MiNet is optimized to gradually align the feature distributions in the spatial domain of micro-expression to that of macro-expression. Given a binary domain label d, indicating if a feature f A ∈ f A a or f A ∈ f A i , the outputs of Ds are used to calculate the spatial domain loss, which can be defined as:

2.4 Guidance fromTemporal Domain

Although spatial stream network can effectively derive spatial patterns from still images, it cannot capture the temporal pattern in consecutive frames. The temporal pattern corresponds to the movement of facial muscles. Although we implicitly encode temporal features by the mean-pooling, the relation between frames is still missing. In order to address temporal variations for videos, we propose an auxiliary task for videolevel features. This is a temporal relation classification task, which predicts the correct relation for temporal features of different sampling interval from both MaNet and MiNet.

Specifically, we first get frame-level features fi = {f 1 i , f 2 i , · · · , f T i } from MiNet. Then, we uniformly sample a frame feature from each s frames with the same temporal interval to generate temporal features vi(s). The temporal features vi(s) with different sampling intervals have consistent context but different playback rates. We use different playback rates to learn slow and fast motion patterns simultaneously. Fast playrate can quickly understand video content, and slow playrate can capture fine details. Similarly, we extract macro-expression temporal features va(s) fromMaNet. We introduce relation classifier Cs to predict the sampling interval for vi(s) and va(s). We set the sampling interval as s = 2k (k = 0, 1, 2, · · · , sc - 1). The ground-truth label is denoted as ys ∈ {1, 2, · · · , sc}, sc is the number of different sampling intervals of the input features. We use cross entropy loss to supervise relation classification task as shown in Eq.3:

where v(s) ∈ {vi(s), va(s)}. The MiNet and MaNet are driven to perceive subtle difference among adjacent frames which is important to learn temporal pattern.

In order to align the temporal features, we also introduce temporal domain discriminator Dt to predict whether the temporal feature v(s) is from micro-expression data or macro-expression data. Given a binary domain label d, indicating if a feature v(s) ∈ vi(s) or v(s) ∈ va(s), the outputs of Dt are used to calculate the temporal domain adversarial loss, as shown in Eq.4:

Through adversarial training with GRL, Lt also contributes to optimize MiNet to align the feature distribution in the temporal domain of micro-expression to that of macro-expression.

Replacing original video-level feature v by aggregating multiple temporal features v(s)(s = 1, 2, 4, · · · , 2sc-1 ), the MiNet can encode temporal relation information.

2.5 Contrastive Loss from Spatial-temporal Domain

Up to now, we only consider minimize the domain discrepancy between micro and macro expression, but neglect the class label, which may lead to misalignment and poor generalization performance. We propose to explicitly take the class label into account and measure the intra-class and interclass discrepancy. The intra-class discrepancy is minimized to compact the features of micro and macro expression within a class, whereas the inter-class discrepancy is maximized to push the features of each class further away from the decision boundary. The contrastive objective functions [Khosla et al., 2020] have achieved excellent performance in recent years by sampling positive pairs and negative pairs. Inspired by contrastive learning framework, for each anchor microexpression sample, we generate many positive pairs and negative pairs from micro and macro expression minibatch.

Specifically, for a minibatch of n randomly sampled microexpression samples {x(j) i , y(j) i }nj=1 , the corresponding randomly sampled minibatch used for training consists of n macro-expression samples {x(j) a , y(j) a }nj=1 . We input them into MiNet and MaNet respectively to get their fused features {z(j) i }nj=1 , {z(j) a }nj=1 . Within aminibatch, let j ∈ {1, · · · , n} be the index of an anchor sample, and generates positive pairs with the same expression labels from micro and macro expression data, generates negative pairs with remaining samples. The contrastive loss takes the following form:

Let sim(w, v) denote the cosine similarity between two vectors w and v, and τ denote a temperature parameter.

During training, for any anchor sample j, all positive pairs in a minibatch contribute to the numerator of Eq.6. This loss encourages the MiNet to give closely aligned features to all entries from the same class in each instance, resulting in a more robust clustering of the feature space for microexpression and macro-expression.

2.6 Overall Loss Function and Optimization

Finally, by combining the losses Lc,Ls,Lsam,Lt and Lcon, we define the general loss as follows:

where λ1, λ2, λ3 and λ4 are the hyperparameters controlling loss coefficients. We use the general loss to update the parameter of MiNet with Adam optimizer. After training, we use MiNet for micro-expression recognition.

3 Experiment

3.1 Experimental Condition

There are three databases commonly used for MER, i.e., CASME II [Yan et al., 2014], SMIC [Li et al., 2013] and SAMM [Davison et al., 2016]. The CASME II [Yan et al., 2014] has 256 micro-expression videos from 26 subjects, with the average age of 22.03 years old at 200 fps. The videos in this database show a participant evoked by one of five categories of micro-expressions: happiness, disgust, repression, surprise and others. The SMIC [Li et al., 2013] database contains 164 micro-expression samples from 16 participants. Each micro-expression is recorded at the speed of 100fps and labeled with three general expression labels: positive, negative and surprise. The SAMM [Davison et al., 2016] database contains 159 micro-expression clips from 32 participants at 200 fps. These participants are from 13 races and the average age is 33.24 years old. Seven micro-expression types are included in the SAMM dataset. They are happiness, surprise, disgust, repression, angry, fear and contempt.

We conduct two experiments on these databases. First, we test our method on the CASME II, SMIC and SAMM databases separately. Accuracy and F1 score are used for evaluation. Second, we test our method on Composite Database Evaluation (CDE) task [See et al., 2019], i.e., samples from all databases are combined into a single composite database based on the reduced expression classes. The distribution of samples and subjects are given in Table 1. Unweighted F1 score (UF1) and Unweighted Average Recall (UAR) are used for evaluation. Leave-one-subject-out (LOSO) cross-validation is used in all experiments.

Two popular lab-collected databases, i.e., CK+[Lucey et al., 2010] and MMI[Pantic et al., 2005] are adopted as macroexpression. The CK+ database contains 327 image sequences of seven expressions: anger, contempt, disgust, fear, happiness, sadness, and surprise. The MMI database includes 205 image sequences with frontal faces of six expression labels: anger, disgust, fear, happiness, sadness and surprise.

In our experiments, expression videos are preprocessed to a fixed length, and every frame is rescaled to a fixed size. We use ResNet18 as the architecture of spatial stream network and temporal stream network. The spatial discriminator and temporal discriminator are three convolutional layers ending with a linear layer outputs a scalar value. The structure of relation classifier is three convolutional layers ending with a linear layer and outputs the prediction of all possible sampling intervals. We use Eq.8 to update the parameter of MiNet with the Adam optimizer, and set λ1 = λ4 = 0.1, λ2 = λ3 = 0.2.

3.2 Experimental Results and Analysis

Recognition on the CASME II, SAMM and SMIC Databases Separately

We conduct ablation experiments to verify the influence of different loss functions, i.e., spatial domain loss Ls, temporal domain loss Lsam+Lt and contrastive loss Lcon on the final recognition performance. As shown in Table 2, We can draw the following observations:

First, adopting one of the introduced losses leads to an improvement comparing with the baseline model only using Lc. Specifically, the accuracy/F1 of Lc + Ls, Lc + Lsam + Lt and Lc + Lcon are 3.6%/4.5%, 4.8%/7.0% and 3.2%/4.7% higher than the baseline on the CASME II database, with CK+ database as macro-expression. The experimental results on the SAMM and SMIC databases show similar trend. Limited by the amount of micro-expression data, deep methods don't have good generalization. However, our method takes advantage of the macro-expression data effectively and transfers them to micro-expression recognition by adopting two auxiliary tasks in spatial domain and temporal domain and contrastive loss in spatial-temporal domain.

Second, the spatial and temporal domain losses achieve better performance than contrastive loss in most cases. For example, Lc +Ls +Lsam+Lt gains 2.4%/3.9%, 2.4%/1.2% and 2.0%/2.2% in accuracy/F1 than Lc + Lcon on the CASME II, SMIC and SAMM database respectively, with CK+ database as macro-expression. Although contrastive loss takes the class label into account and measures the intraclass and inter-class discrepancy, it is difficult to choose a suitable distance metric.

Finally, our method combines the strengths of the four introduced loss functions and achieves the best performance. Specifically, the accuracy/F1 of our method is 7.6%/11.2%, 9.1%/10.6% and 6.6%/7.1% higher than the baseline on the CASME II, SAMM and SMIC database, with CK+ database as macro-expression. This indicates that the different guidance will not cause the inter-domain discrepancy. Two auxiliary tasks only align overall feature distribution and don't consider the distribution of each category, but contrastive loss takes the class information into account.

Recognition of CDE Task

CDE task is proposed to evaluate micro-expression recognition on composite database including CASME II, SMIC and SAMM. Tasks on cross databases have both pros and cons. The valuable training data will be more sufficient for training and this is essential for micro-expression recognition since the available data are too scarce. But since composite database comprises samples collected from different environment and subjects, training across different databases will suffer from domain shift problem. Since our method takes advantage of the macro-expression databases for micro-expression recognition, and adopts two auxiliary tasks to learn domain-invariant features, our method can avoid domain shift problem and take full use of composite database. As shown in Table 3, the UF1/UAR of our method is 8.2%/7.8% and 6.2%/6.2% higher than the baseline on the CDE task, with CK+ and MMI as macro-expression dataset.

3.3 Comparison with RelatedWorks

We compare our framework with related works. These methods are: 1) LBP-TOP [2016], LBP-SIP [2014], STLBPIP [2015], STCLQP [2016], HIGO [2017], FHOFO [2017] and Bi-WOOF [2018], which are handcraft feature methods, 2) OFF-Apex [2019], Boost [2019], DSSN [2019], AUGACN [2020], Dynamic [2020], Graph-TCN [2020], Shallow [2019], Dual [2019] and Capsule [2019], which are deep feature methods, 3) Neural [2019] and MicroNet [2020], which are macro-expression assisted methods.

From Table 4, we can see that our framework exceeds most handcraft feature methods in almost every evaluation indicators. Our framework achieves nearly 21.1%, 16.4% and 18.4% increases in accuracy, 14.9%, 15.8% and 36.7% increases in F1 score compared to the best results of handcraft feature methods, i.e., Bi-WOOF on the CASME II, SMIC and SAMM database. Due to short duration and low intensity of micro-expression, these handcraft feature methods can't capture the details of facial appearance. Our method also outperforms the best results of deep feature methods, i.e., Graph-TCN by 5.9%/3.4% and 2.7%/6.5% on the CASME II and SAMM database of accuracy/F1. Graph-TCN applied video motion magnification to magnify the intensity of microexpression, but this operation would introduce noise to microexpression data. In this paper we introduce auxiliary tasks to make micro and macro expression samples produce similar feature distributions in spatial and temporal domain.

As shown in Table 5, our method gains higher results on CDE task. Because the CDE task has greatly increased microexpression training data, deep feature methods (i.e., Shallow, Dual and Capsule) can't handle the difference between databases very well. As a result, these methods perform well on the CASME II database, but get poor results on the SMIC and SAMM databases. However, our proposed framework adopts two auxiliary tasks to learn domain-invariant features from macro-expression data, which avoid domain shift problem and take full use of composite database.

Compared with the macro-expression assisted methods, i.e., Neural and MicroNet, our proposed method exceeds their results in both composite and single databases. Neural used EMR to reduce the gap between micro and macro expression visually. This preprocessing can't guarantee the similarity of micro and macro expression features, then directly training on a fusion of micro and macro expression database can't generate appropriate expression features. MicroNet can't be trained end-to-end, which impairs the performance of overall framework. Neural and MicroNet only capture static structure patterns from spatial domain, and ignore temporal patterns. However, our method adopts a temporal relation classification task to learn dynamic movement patterns.

4 Conclusion

In this paper we propose a macro-to-micro transformation model which transfers spatial and temporal pattern existed in macro-expression to micro-expression recognition. In order to take advantage of macro-expression data, we introduce adversarial learning to align the spatial features of MiNet and MaNet. In temporal domain, our proposed auxiliary task predicts temporal relation. Through this task, MiNet can learn the dynamic movement relationship of the muscles from MaNet. Finally, the proposed contrastive loss encourages the MiNet to give closely aligned features to all entries from the same class in each instance. Experiments on three benchmark databases demonstrate that our framework outperforms the state-of-the-art micro-expression recognition methods.

Acknowledgments

This work was supported by the National Key R & D program of China 2020YFC2007700, project from Anhui Science and Technology Agency 1804a09020038.