MMNet: Muscle Motion-Guided Network for Micro-Expression Recognition

1 Introduction

Facial expressions are an essential carrier for spreading human emotional information and coordinating interpersonal relationships. Most of the expressions that we see in our daily life are macro-expressions. However, spontaneous, brief, and subtle micro-expressions (MEs) can reveal people's true feelings when people try to hide their real emotions under certain conditions [Ekman, 2009], which makes MEs applicable to many areas such as criminal interrogation, clinical diagnosis, and human-computer interaction. Different from macro-expressions, MEs are usually accompanied by tiny facial muscle motions and last less than half a second (usually

∗Corresponding author.

1/25 to 1/3 second), which makes micro-expression recognition (MER) task very difficult for humans, and even more difficult for computers.

According to the features extraction methods, MER techniques can be roughly divided into two categories: handcrafted approaches and deep network-based approaches. For the former, histogram of oriented gradient (HOG), histogram of optical flow (HOOF), and local binary pattern-three orthogonal planes (LBP-TOP) are often used to extract ME features. Le Ngo et al. learnt temporal and spectral structures with sparsity constraints by processing the LBP-TOP [Le Ngo et al., 2016]. Li et al. adopted the histogram of image gradient orientation-TOP (HIGO-TOP) for MER [Li et al., 2017]. Happy et al. proposed a fuzzy-based HOOF (FHOOF) feature extraction technique that only considers the muscle motion direction for MER [Happy and Routray, 2017]. However, due to the short duration and the inconspicuous motion of MEs, handcrafted features are often unable to robustly represent the differences between different micro-expressions, which is detrimental to MER.

In recent years, with the development of deep learning technology, more and more researchers have achieved promising results by designing deep neural networks (DNNs) to handle MER tasks. For example, Gan et al. introduced a feature extractor that incorporates both the handcrafted (i.e., optical flow) and data-driven (i.e., convolutional neural networks, CNNs) features [Gan et al., 2019]. Song et al. proposed a three-stream convolutional neural network (TSCNN) to recognize MEs by learning ME-discriminative features in three key frames of ME videos [Song et al., 2019]. Xie et al. developed a MER approach by combining action units (AUs) and emotion category labels [Xie et al., 2020], meanwhile Lei et al. designed a graph temporal convolutional network (Graph-TCN) to extract the local muscle motion features of MEs. Xia et al. devised a framework that leverages macro-expression samples as guidance for MER [Xia et al., 2020]. Among the latest results, Xia et al. designed a macroto-micro transformation framework by two auxiliary tasks from the spatial and temporal domains, respectively [Xia and Wang, 2021].

Although the above methods have gradually improved the performance of automated MER algorithms in recent years. Nevertheless, most of them directly input the original video frames or the handcrafted features of the video frames into DNNs to extract features of MEs, which makes DNNs easy to lead into the identity information of the samples. Obviously, identity information that has nothing to do with expression is harmful to facial expression recognition (FER) task. This problem may have little impact on the macro-expression recognition task with abundant training data [Li and Deng, 2020]. However, due to the extremely cost-consuming of collecting and labeling micro-expression data, we still do not have a large-scale micro-expression dataset comparable to the macro-expression datasets (e.g., AffectNet [Mollahosseini et al., 2017]). MEs are mainly determined by the position of facial muscle motion and the muscle motion pattern (e.g., the slight upturn of the lips corner on both sides is likely to indicate happiness). So, the key of MER is to learn the position and pattern of facial muscle motion rather than directly learn from the entire video frames.

As shown in Figure 1, we propose a new two-branch MER paradigm to deal with the two key factors mentioned above, which extracts muscle motion-pattern features from the difference between the onset frame and the apex frame through the main branch, and generates facial position embeddings from the onset frame only through the subbranch. We also give a specific realization of the proposed two-branch MER paradigm, namely muscle motion-guided network (MMNet), which mainly focuses on learning the position (e.g., lips corner and upper eyebrow) and motion patterns (e.g., raising and lowering) of facial muscle motions. Specifically, we learn the features of micro-expression video sequences by a twobranch framework. First, we introduce a new continuous attention (CA) block to learn the patterns of facial muscle motions. CA block can pay attention to the location of the motion and extract features related to motion patterns. Second, we devise a position calibration (PC) module based on vision transformer (ViT) [Dosovitskiy et al., 2020] to add robust facial position information to the learned motion-pattern features. It is worth noting that the main branch of our MMNet only models the difference between the onset frame and the apex frame, which reflects the muscle motion on the face and makes the model less affected by the identity information.

The contributions of our work are summarized as follows:

• We propose a novel two-branch MER paradigm, which extracts the muscle motion-pattern features and facial position embeddings through the main branch and subbranch, respectively. Then, the two kinds of features are fused at the end of the network for classification.
• We devise a muscle motion-guided network (MMNet) to implement the above two-branch MER paradigm. The main branch for extracting the motion-pattern features is composed of the proposed continuous attention (CA) block, and the subbranch for generating the position embeddings is realized through the designed position calibration (PC) module.
• Our MMNet outperforms the state-of-the-art approaches by a large margin on three popular micro-expression datasets (i.e., CASME II, SAMM, and MMEW). Extensive experiments demonstrate the effectiveness of the proposed MMNet.

2 Method

In this section, we detail the two branches of our proposed MMNet for MER.

2.1 ProblemFormulation and Overview

Existing MER methods often focus on designing the structures of deep networks to improve the recognition accuracy, but usually neglect to find better ways to utilize the video frames of MEs. Most approaches input the original video frames with identity information into a single-branch network [Li et al., 2020; Lei et al., 2020], or extract handcrafted features (e.g., optical flow or LBP) from the frames and then fuse them by a multi-branch network [Liong et al., 2019]. Since the input for the main branch contains the identity information of the sample to varying degrees, the network is likely to learn identity related features that have nothing to do with the MER tasks, especially when ME data are insufficient. To tackle this problem, we raise a novel two-branch MER paradigm. The main branch is designed to deal with the motion pattern, and the light-weight subbranch extracts facial position embeddings from the low-resolution onset frame that mainly contains the facial position information without any expression related information. In this way, even if the lowresolution onset frame contains certain identity information, since it is only fed into the light-weight subbranch, the whole network can still focus on learning the motion pattern rather than identity.

As seen in Figure 2, our proposed MMNet mainly consists of two branches. The main branch extracts the motion-pattern features with the difference between the apex frame and the onset frame as the input, and the subbranch is used to generate the facial position embeddings with low-resolution onset frame as the input. Finally, the facial position embeddings are added to the motion-pattern features for mapping the motion pattern to specific face areas.

2.2 Continuous Attention Block

Compared with macro expressions, MEs tend to have smaller muscle motions and more local active areas, which make it difficult for the traditional attention modules to accurately focus on the subtle facial muscle motions. To alleviate this, some existing works try to model the relationship between muscle motions and MEs [Xie et al., 2020] with the help of action unit (AU) labels [Ekman and Friesen, 1978]. However, it is still a huge challenge to obtain precise attention maps without introducing extra supervision.

To address the above issue, we devise a continuous attention block by introducing the attention maps of the previous layer as the prior knowledge to generate the attention maps of the current layer. Inspired by the spatial attention module of convolutional block attention module (CBAM) [Woo et al., 2018] depicted in Figure 3(a), we utilize both max-pooling outputs and average-pooling outputs to calculate the spatial attention maps. As shown in Figure 3(b), we make the attention maps of the previous layer as the prior knowledge to obtain the attention maps of the current layer, and use a smaller convolution kernel (i.e., 1×1) to obtain more local attention maps. Formally, the CA module can be defined as,

with

where Mi is the CA module of the ith CA block that expects to pay attention to the muscle motion areas and Attni-1 is the attention maps of the (i - 1)th layer. F i conv ∈R2C×H×W denotes the features extracted by the first two convolutional layers of the ith layer as the input of CA module. σ characterizes the sigmoid function. f i1×1 and f i3×3 represent a convolution operation with the kernel size of 1 and 3 from the ith layer, respectively. PMc(F i conv )∈R1×H×W and PAc(F i conv )∈R1×H×W separately describe the max-pooled features and average-pooled features across the channel. PM indicates the max-pooling operation on the attention maps of (i - 1)th layer to match the size of current layer attention maps, while ⊗ means the element-wise product for introducing the attention maps from last layer as prior knowledge. F i∈RC×2H×2W stands for the input of the ith CA block.

By associating the attention mechanism between adjacent layers, the CA block can gradually and robustly focus on the areas that have subtle motions, instead of focusing on different areas of the face in different layers, which may make networks learn both ME related areas and unrelated areas. We use four CA blocks to constitute the main branch of MMNet to learn the subtle muscle motion-pattern features FM with the size of 512×14×14. The CA block which consists of the CA module and two convolutional layers can be formulated as,

where CA represents the proposed continuous attention block and ⊛ stands for broadcast element-wise multiplication, which means each channel of F i conv ∈R2C×H×W will be multiplied by the spatial attention maps Mi (F i conv ,Attni-1 )∈R1×H×W to pay attention to the region of interest. The ablation experiments and visualization results in Figure 4 demonstrate that our CA block can help to concentrate on accurate ME related areas. As shown in Figure 2, we take the difference between the apex frame and the onset frame as the input of the main branch to learn the motion-pattern features.

2.3 Position CalibrationModule

Due to the diverse appearances of different people in the micro-expression datasets, it is hard to strictly align all the faces because of various interpupillary distances, dissimilar nose sizes, etc. Therefore, the same face areas may correspond to different pixel positions of the image, which make it hard for the network to learn exactly where the subtle motion occurs. In order to accurately add position information to the motion-pattern features extracted by the main branch, we propose a position calibration module as the subbranch of MMNet to generate the facial position embeddings for mapping motion-pattern features to specific areas of the face.

Since the relative positions of the facial features are physically determined (e.g., the nose is usually located below the middle of the two eyes), modeling long-distance dependencies can effectively help locate the positions of various parts of the face and generate robust position embeddings. Recently, ViT applies self-attention mechanism to model longdistance dependencies and has achieved promising results on image classification tasks, while the convolutional neural networks (CNNs) often need many convolutional layers to obtain a global receptive field, which is not conducive to modeling facial position information. So, we introduce the PC module based on a shallow ViT. As shown in Figure 2, we utilize the difference between the apex frame and the onset frame to learn the motion-pattern features FM and the lowresolution onset frame to learn the facial position embeddings Epos. Since we only need to learn the locations of salient areas (e.g., eyes, mouth, and nose), instead of the detailed texture related to the identity of samples (e.g., wrinkles and skin tone), we scale the onset frame to the same size of 14×14 as the input of subbranch matching the size of FM. Then, we reshape the scaled onset frame into a sequence of 196 flattened 2D patches Ip with the size of 1×1×3, and map them through a trainable linear projection to get patch embeddings Ep with 512 dimensions, which matches the channel dimension of FM. After adding the position embeddings of ViT to retain the position information as done in [Dosovitskiy et al., 2020], these patch embeddings are sent to the transformer encoder to learn the relationship between patches. Finally, the output of ViT with the size of 196×512 is reshaped into 512×14×14 to get Epos for position calibration. The position embeddings Epos are then added to the motion-pattern features FM for mapping the motion pattern to specific face areas for MER.

3 Experiments

To verify the effectiveness of our MMNet, we conduct extensive experiments on three popular micro-expression datasets including CASME II [Yan et al., 2014], SAMM [Davison et al., 2016], and MMEW [Ben et al., 2021]. In this section, we first introduce these three datasets and the implementation details. Then, we explore the performance of the two branches of MMNet, respectively. Finally, we compare our method with several state-of-the-arts.

3.1 Datasets

CASME II [Yan et al., 2014] contains 256 micro-expression videos from 26 subjects with a cropped size of 280×340 at 200 fps. Consistent with most of previous methods, only samples of five prototypical expressions, i.e., happiness, disgust, repression, surprise, and others, are used. SAMM [Davison et al., 2016] has 159 micro-expression clips from 32 participants of 13 different ethnicities at 200 fps. Five expressions (happiness, anger, contempt, surprise, and others) are utilized for experiments. MMEW [Ben et al., 2021] includes both macro- and micro-expressions sampled from the same subjects for researchers to explore the relationship between them. It contains 300 MEs and 900 macro-expression samples with a larger resolution (1920×1080) at 90 fps. We use four expressions (happiness, surprise, disgust, and others) for ablation studies. Table 1 shows the data distributions for each class of CASME II (five classes), SAMM (five classes), and MMEW (four classes), while Table 2 gives the data distributions for each class of CASME II (three classes) and SAMM (three classes). First, we conduct ablation studies on these three datasets separately to verify the effectiveness of the CA block and PC module. Second, since the MMEW dataset was just released in 2021, there are currently fewmethods for comparison on this benchmark. So, we mainly compare MMNet with other state-of-the-art approaches on CASME II and SAMM datasets. Consistent with most of previous works, leave-one-subject-out (LOSO) cross-validation is employed in all the experiments, which means every subject is taken as a testing set in turn and the rest subjects as the training data. For all the experiments, the accuracy and F1-score are used for performance evaluation.

3.2 Implementation Details

In our experiments, we use Dlib to detect the landmark points on the face and crop the images according to these landmark points. The cropped images on all the datasets are resized to 224 × 224. To avoid overfitting, we randomly pick a frame from four frames around the labeled onset and apex frames as the onset frame and apex frame for training. The horizontal flipping, random cropping, and color jittering are also employed. We use four CA blocks to constitute the main branch and a shallow ViT to build the subbranch. At the training stage, we adopt AdamW to optimize the MMNet with a batch size of 32. The learning rate is initialized to 0.0008, decreased at an exponential rate in 70 epochs for cross-entropy loss function. All the experiments are conducted on a single NVIDIA RTX 3070 card with PyTorch toolbox.

3.3 Ablation Studies

Effectiveness of the Two Branches in MMNet. To verify our CA block and PC module, we set a ResNet consisting of four building blocks which is used in ResNet18 as the baseline model for ablation studies. Then, we separately replace the building blocks of ResNet18 with the CA block and add the PC module to compare the proposed model with the baseline. As shown in Table 3, the CA block and PC module both can significantly improve the performance. Finally, when we use both the modules to build our MMNet, the results exceed the baseline by 7.23%/10.94%, 6.61%/9.46%, and 5.70%/7.14% on CASME II, SAMM, and MMEW datasets in terms of accuracy/F1-score.

Comparisons between Continuous Attention and Independent Attention. To illustrate the effectiveness of introducing attention maps of the previous layers as prior knowledge on generating precise attention maps for MER task, we compare the proposed continuous attention module with traditional independent attention module (i.e., spatial attention module of CBAM shown in Figure 3(a)), which generates the spatial attention maps independently in each layer. It is worthy to note that both settings add the PC module to generate the position embeddings. As shown in Table 4, the performance of CA module on the three datasets are significantly better than the independent attention module.

Impact of the number of layers and heads of PC module. Vision transformer encoder consists of NL identical layers. The multi-head self-attention in each layer enables the model to decompose the information into NH representation subspaces and jointly capture discriminative information at different positions. We explore the effects of different layer values NL and the number of heads NH on CASME II and SAMM datasets. Table 5 shows the performance comparison of different hyper-parameter settings of our method. We observe that smaller NH and NL tend to achieve better results. We think this is because larger models are more prone to overfitting. Therefore, we set the number of self-attention heads NH and the number of transformer encoder layers NL to 2 and 4 by default.

3.4 Comparison with State-of-the-arts

We also compare our MMNet with several state-of-the-art methods. From Tables 6 and 7, we can see that our MMNet outperforms the best results of previous methods on every evaluation indicators. Specifically for three-class MER

tasks, our method outperforms GACNN [Kumar and Bhanu, 2021] by 1.5%/2.73% and 5.85%/7.99% on SAMM and CASME II with respect to accuracy/F1-score. As for five classes, our MMNet exceeds SMA-STN [Liu et al., 2020] by 2.94%/2.58% and 5.76%/7.3% on SAMM and CASME II in terms of accuracy/F1-score.

3.5 Visualization

In order to prove that our proposed continuous attention module can pay attention to the movements of tiny facial muscles, we visualize some attention maps of the first CA block in Figure 4. It can be easily seen that CA module can generate attention maps to help the network focus on where the facial muscle moves accurately. Specifically, the CA module concentrates on the upper end of the eyebrows for surprise samples and the corner of the mouth for happiness samples.

4 Conclusion

In this paper, we develop a new two-branch MER paradigm and make a realization of the new paradigm, called MMNet. Specifically, the main branch based on the proposed CA block focuses on learning motion-pattern features from the difference between the onset and apex frames, while the subbranch based on the PC module concentrates on generating facial position embeddings for position calibration. Extensive experiments illustrate that our MMNet outperforms state-of-the-arts by a large margin on CASME II and SAMM datasets.

Acknowledgements

This work was supported by the JKW Research Funds under Grant 20-163-14-LZ-001-004-01. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.