Fusion of Granular-Ball Visual Spatial Representations for Enhanced Facial Expression Recognition

1 Introduction

As artificial intelligence technologies advance at a rapid pace, understanding human emotions through facial expressions has become paramount for creating intelligent systems that can interact naturally with humans. Facial expression is one way for us to express our emotions without words, and if machines can read these "silent signals", they can engage with us more effectively. According to researches [Taheri et al., 2013], the variation in facial expressions is caused by the brain transmitting emotional signals, and the resulting facial muscle movements lead to morphological changes in the organs of the facial component (hereafter referred to as components), thus forming different facial expressions. As shown in Fig. 1, when someone is angry, his eyes will obviously widen, pupils will become smaller, eyebrows will be depressed, etc. These forms of expression are not only intuitive visual expressions, but also have internal spatial structural differences, such as changes in relative positions between different components, changes in the shape of individual components, etc. However, the vast majority of researches ignore the exploration of spatial information or fails to use spatial information effectively for facial expression recognition tasks.

* Corresponding author.

Many proposed deep learning-based expression recognition methods [Psaroudakis and Kollias, 2022; Roy and Etemad, 2023; Li et al., 2022] have achieved good results. However, most current recognition methods focus entirely on extracting the visual representation of expression images to achieve expression recognition tasks [Wu et al., 2023b]. This type of method focuses solely on extracting visual representations and directly discards the spatial structure information of facial expressions. After the emergence of graph neural network (GNN), a considerable number of researchers applied it to the field of expression recognition, hoping to use the advantages of graph neural network to extract spatial structure information of visual representation [Li et al., 2023]. However, the inputs transmitted to GNN by these methods are hidden features of the same image passing through different network layers, or hidden features of different patches passing through the same network layer. What these methods have in common is that the features processed by GNN are non-graph structured data or non-source data that have been extracted from images through the backbone of CNN-based or Transformer-based. As mentioned earlier, image itself has very limited spatial information, and a lot of spatial information has been lost after backbone.

With the advent of facial landmark localization, as shown in Fig. 1c, more researchers try to use landmark information to extract spatial structure information in expressions [Zhao et al., 2024; Wu and Cui, 2023]. However, accurate facial landmarks require not only labor-intensive labeling to obtain but also provide limited spatial information that is highly correlated with the quality of label. As shown in Fig. 1b, we propose a novel and more effective way to represent spatial structure information of expressions by importing granular balls method [Xia et al., 2023], which represents expression as graphs with rich spatial information.

Compared to images composed of tens of thousands of pixel points, graphs consist of a finite number of edges and points. In addition, the edge-point structure makes graphs richer in spatial information while the number of data points is reduced. Granular balls representation is a data representation method that reduces redundant data and improves model generalization. Applying it to 2D image data to process pixel points [Xia et al., 2023], we are able to give more spatial structure to expressions while ensuring the quality of expression images. In order to solve the problem of how to effectively utilize spatial information to improve the accuracy of FER, we propose a novel FER method based on Component Separation and Space Bootstrap Fusion (CS-GBSBF), which mainly consists of Represent Extraction Network (REN), Represent Separation Network (RSN) and Represent Fusion Network (RFN).

Specifically, input data are divided into image and graph streams, then the data are first fed into the two backbone of the REN respectively. For images and graphs, the backbone is used to extract basic visual and spatial representation features. Then, two types of features are input to Visionrepresent Separation Network (VRSN) and Space-represent Separation Network (SRSN) in RSN, which effectively separate the basic features into component representation features. Next, component features are input into RFN, which uses component spatial representation features to bootstrap the fusion of component visual representation features. In particular, bootstrap alignment loss is developed to solve the nonalignment problem that occurs when guide fusion of component features. Finally, a classify head network is employed for expression classification.

Our main contributions can be summarized as follows.

• A novel CS-GBSBF method is proposed to perform FER. In CS-GBSBF, granular balls representation is innovatively applied to expression images and imports a
• new spatially structured representation for the 2D visual expression domain.
• We innovatively propose a novel FER paradigm based on component representation, where the overall facial expression is separated into different components. RSN is developed to extract the visual representation features of components along with the spatial representation features. In RFN, bootstrap alignment loss is developed to solve the problem of unaligned fusion between components caused by RSN.
• Our method is evaluated on multiple popular real and wild databases compared with a variety of state-of-theart methods in recent years and achieves superior results. In particular, CS-GBSBF achieves recognition accuracy of 97. 34% and 96. 74% on the databases Oulu-CASIA and SAMM, respectively.

2 RelatedWorks

2.1 Facial Expression Recognition

Facial expression recognition (FER) is a complex and practical CV task that has gained significant attention recently.

With the substantial advancements over the past decades, deep learning has shown extraordinary performance in extracting visual representational features of expressions, making the application of deep learning models to FER tasks increasingly popular. Since then, an increasing number of researchers have used deep learning to extract more effective visual representational features [Psaroudakis and Kollias, 2022; Roy and Etemad, 2023]. Sun [Sun et al., 2023a] propose a novel Feature Decomposition and Reconstruction Learning method for FER. In the process of research, successive researchers have discovered that there is spatial structure information between facial components. The graph neural network model was imported to extract the spatial information, and different graph structure data fed into GNN were designed [Liu et al., 2024; Li et al., 2023]. Kim [Kim et al., 2023] propose that a face graph is constructed by combining the attention map with face patches and then is fed to GCN.

2.2 Granular-ball Computing

Granular ball is a method for data representation, which is inspired by the "large scale first" cognitive mechanism. Granular balls represent the data using multiple scales granular balls, which greatly reduce the amount of data while preserving the quality of the original dataset. In recent years, granular balls has made significant strides in many fields, such as clustering [Cheng et al., 2023], classification [Xie et al., 2024], graph generation [Xia et al., 2023], etc. Zhang et al. [Zhang et al., 2023] proposed GBRS based on incremental granularity computation for better classification tasks. Quadir et al. [Quadir and Tanveer, 2024] proposed a granular ball twin support vector machine to deal with significant challenges in the TSVM field.

In this paper, we import the method of granular balls representing images [Xia et al., 2023] to the field of FER by structuring expression images as graphs. Image is one kind of high-density point-integrated data, while graph is the kind of sequence-structured edge-point data. Compared with image, graph has richer spatial information, such as relative position, spatial topology, angular direction and so on. Particularly, the conversion from image to graph is highly consistent with the principles and advantages of granular balls representing data.

3 ProposedMethod

3.1 Overview

The proposed CS-GBSBF method consists of a Represent Extraction Network (REN), a Represent Separation Network (RSN), a Represent Fusion Network (RFN) and a Classify Head Network (CHN). An overview is shown in Fig. 2.

Given a batch of facial expression images, we firstly use granular balls representation to generate the corresponding graphs of the images. For the too large graphs that are difficult to be processed, the model will use Graph Down Sample (GDS) to downsample graphs to obtain the graphs that are in appropriate size. Then, images are input into Vision-represent Extraction Network (VREN) in REN to extract basic visual representation features, while graphs are input into Spacerepresent Extraction Network (SREN) in REN to extract basic spatial representation features. In order to align the regions of features extracted by VREN and SREN, we set up an attention steering module that employs the attentional output of VREN to correctly guide the attention regions of SREN. Next, basic visual and spatial representation features will be fed to VRSN and SRSN, respectively, in order to separate and enhance the basic visual/spatial features and then gain the component visual/spatial representation features. Then, the two types of component features will be fed into RFN, which uses component spatial features to guide the fusion of the component visual features. Finally, CHN, equipped with a categorical linear layer, performs FER on the fused features.

3.2 Granular-ball Represent

With the advent of GNN, there is a growing number of methods for representing non-graph data as graph-structured data.

In the case of expression images, basically all the representations follow the same paradigm, which first extracting the features of the same sized region block using visual network model, and then constructs graph-structured data to expect to enrich spatial information of data. Unlike previous methods, we propose to utilize granular balls [Xia et al., 2023] to represent expression images xij , that is:

where GBR is the process of granular balls representation images and xgj is the graph after representation.

Compared to previous methods, our method preserves rather than produces spatial information from the source of images rather than the intermediate data, and extracts spatial information such as the shape and density of granular balls, as well as the relative positions and relative distances between granular balls, in a multi-granularity rather than a single-size patch manner. The basic idea of GBR algorithm is to divide the image into matrix regions of different sizes with the criterion of ensuring that the pixel values of all positions within each matrix are as similar as possible, as shown in Fig. 1b. Each of these matrices is treated as one point in graph, and the region intersecting matrices are treated as if there is one edge between the matrices.

The number of nodes of graph represented by granular balls method is highly dependent on the resolution and color complexity. An image with resolution 384 × 384 may extract nearly 10,000 nodes, some of which are located in the hair, clothing, and other non-expression related areas, which will import noise nodes. We use downsample to merge nodes to reduce noisy nodes while keeping the effective spatial information. Unlike the edge-focused downsample method proposed in [Xia et al., 2023], we improve the downsample method with the basic idea that if a node and another node with minimal distance is merged for the downsample operation. More specific details can be viewed in Appendix1 A.

1 The appendix material version is available at https://github.com/Lsy235/CS-GBSBF/tree/main/supplyMaterials.

3.3 Represent Extraction Network

Given the j-th expression image xij and the j-th corresponding graph xgj , VREN will extract the basic visual representational features Fvb ∈ R1×De in xij . As well as SREN will extract the basic spatial representational features Fsb ∈ R1×De in xgj , which can be formulated as:

where Lvl denotes the l-th network layer of the VREN and Lsl denotes the l-th network layer of the SREN. N indicates the number of layers in the backbone network.

Since REN contains dual data processing streams for VREN and SREN, there are bound to be unaligned problems in the feature regions where VREN and SREN focus on. To solve this problem, we set Attention Guidance Module (AGM) in REN, which is expected to be used for the attention output of VREN to guide the attention of SREN to the corresponding region. Specially, the attention output of the l-th layer of VREN is summed with the output of the l-th layer of SREN after a layer of fully connected layer transformation, which can be formulated as:

where outsl ∈ R1×De and outvl ∈ R1×De denote the output of the l-th network layer of SREN and VREN, respectively. FC indicates a fully connected layer.

Images have better location information, while graphs have more information of relative location that can reflect spatiality. We use VREN to guide SREN. When l takes a later value, the attention of VREN is more obvious and the effect of guidance is better. In the realization, the value of l is set to N.

3.4 Represent Separation Network

Fvb and Fsb extracted by REN which are broadly focused on all the components of the face but not specific to one, as shown in the attention heat map in Fig. 3. As mentioned in the opening section, facial expression consists of a combination of variations of components. Therefore, in order to extract the component visual representation features Cvk ∈ R1×Ds with the component spatial representation features Csk ∈ R1×Ds , we propose RSN to separate Fvb and Fsb , which can be formulated as:

where K denotes the number of components separated by the overall face. sigmoidk indicates the sigmoid activation function for the k-th component and GeLUk indicates the GeLU activation function for the k-th component. In addition, each component has its own network layers.

We separate the overall face into K components for the expectation that the representational features of each component correspond to the representational features of an unoverlapped region of the face. As shown in the attention heat map in Fig. 3, Fvb and Fsb mainly focus on the overall face. While Cvk and Csk in Eq. (5) appear to split the overall attention and gradually focus on different face regions. Next, we employ sigmoid function to augment the attention. sigmoid often plays an important role in anomaly detection and logistic regression binary classification, while the meaning of Cvk and Csk are whether or not to pay attention to one region of the face, which belongs to the same binary classification task. In the process of continuous supervised training, Cvk and Csk tend to focus more and more on the regions where the differences between samples are large. The most obvious regions where the differences between expressions are at the region of facial component, such as mouth, eyes, eyebrows and so on. After the model has fully converged, the regions of attention for both Cvk and Csk are distributed near the facial components.

3.5 Represent Fusion Network

For the task of FER, not all Cvk and Csk are beneficial for recognition. At the same time, expression recognition does not require observation of the full components [Barros and Sciutti, 2021]. As shown in Fig. 1, based on the changes in the eyebrows, eyes and mouth alone, we can fully deduce that the one is in an angry spirit. Only the components that undergo change are useful, the more change, the more useful. However, due to the differences between samples, the degree of usefulness of component change for expression recognition can be inconsistent across samples. Then, to learn the fusion feature Fu ∈ R1×Ds with high generalization, adaptive weights reflecting the degree of usefulness must be designed to guide feature fusion. Csk extracted by SRSN is fully compatible. The spatial structure information can well reflect the degree of change that has occurred in component. When one component has a relatively large spatial structure transition, then that component must be an important representation of that individual's expression. As shown in Fig. 2, we first multiply the transposed Csk with Cvk in vector counterparts, and the result obtained is passed through softmax function. Then, the weight W ∈ R1×K , which reflect the degree of component change, are obtained, that is:

where softmax denotes the softmax activation function.

Csk with Cvk are learned by REN and RSN based on samples. The differences between the samples make Csk with Cvk also have variations. W in Eq. (6) will adaptively modify the weight values for different samples, which makes Fu have higher generalization. Eventually, W is used to guide the fusion of visual representation features to obtain Fu. Specially, matrix multiplication of Wk ∈ R1×1 with Cvk leads to Fu, which can be formulated as:

where Fu will be fed to CHN for the task of FER.

Bootstrap Alignment Loss. W is particularly important in the whole process of feature fusion in RFN. As well as in Eq. (6), although we have given an explicit inference formula, we have neglected the question of whether the components indicated by Cvk and Csk are the same component. In other words, although REN is designed with AGM to guide attention region alignment, whether Cvk and Csk after RSN separation can still be aligned. It is obvious that RSN does not understand alignment. When the RSN separates Fvb and Fsb of different samples, there is a high probability that the resulting Cvk and Csk cannot be aligned. To solve the problem of representation alignment, we propose Bootstrap Alignment Loss LBA. The ideal meaning of W is the degree of component change. In addition, the larger the degree of component change is, the more effective the corresponding Cvk is for expression recognition. Then, we directly perform facial expression recognition based only on each Cvk . Next, we can obtain K recognition accuracies, which are denoted as ACCs ∈ R1×K . It can be formulated as:

(b) Space-represent Separation Network

Figure 3: The proposed (a) Vision-represent Separation Network (VRSN) and (b) Space-represent Separation Network (SRSN). More details of VRSN and SRSN can be found in Sec. 3.4.

where acc denotes the computational function of recognition accuracy and y denotes the label of expression images.

At this point, ACCs are also expressed as the extent to which each Cvk contributes to the recognition of that expression. In addition, the values of ACCs and W are at the same order of magnitude, ranging from 0 to 1. Then, we calculate the KL divergence of ACCs and W to see if the distributions are consistent. If there is agreement, then it is numerically proven that W has the same distribution as ACCs. The meaning of ACCs for FER task does not alter and then the only thing that can be proven is that W reaches the desired meaning during the process of training. Mathematically, LBA is formulated as:

where KL(..) indicates the function of KL divergence.

3.6 Joint Loss Function

In the proposed CS-GBSBF, REN, RSN, RFN and CHN are jointly trained in an end-to-end manner. The whole network minimizes the following joint loss function:

where Lcls and LBA represent the classification loss and bootstrap alignment loss. In this paper, we use the crossentropy loss as the classification loss. λ denotes the regularization parameter. By minimizing the joint loss, CS-GBSBF is able to extract discriminative represent features for FER.

4 Experiments

4.1 Databases and Evaluation Setups

The databases involved in the experiments include AffectNet-8, CAER-S, RAF-DB, Oulu-CASIA, CK+, SFEW2.0, FER-2013 and SAMM, as shown in Table 1.The evaluation metric of Accuracy and UF1 are selected. UF1, named unweighted f1-score, is one metric that is more effective in evaluating on multicategorical unbalanced databases, and is commonly used in the field of microexpression.

For the database divided by the original test database, we keep it constant. And for the undivided database, we follow the rules most methods [Gan et al., 2019] adopt for classification.

4.2 Experimental Settings

For each database, all the facial images need to be represented by granular balls, followed by graph downsample operation to obtain the input graphs. During the process of training, images are further resized to 224 × 224 and then a random crop is applied for data augmentation. During the test process, images are resized to the size of 224×224 and then fed into the trained model along with graphs. The CS-GBSBF method is implemented with the Pytorch toolbox, employing swinT-base [Liu et al., 2021] as the backbone of VREN and utilizing GCN to build the backbone of SREN, where the swinT-base is pre-trained on the ImageNet-1K database.

The dimensions of both Fsb and Fsb are Rb×1024 and Cvk and Csk have dimensions of Rb×128 . Based on extensive experiments in the validation set, the value of λ in Eq. (10) is empirically set to 0.1, while the value of the hyperparameter K is set to 9. We train CS-GBSBF in an end-to-end manner with one single NVIDIA GeForce RTX 4080 SUPER for 40 epochs, and the batch size for all databases is set to 16. Then our model is trained using the Adam algorithm with an initial learning rate of 0.0001, weight decay = 0.01.

4.3 Ablation Studies

The ablation studies in this section are recorded in the validation set. The ablation studies set up in this subsection focus on the important modules of the proposed design (AGM, SRSN, LBA), variation in the value of the component number K, and the weight value λ. The effect of each module in the proposed method is explored through ablation studies on the validation set. Ablation studies for the other model parameters are shown in Appendix B.

Influence of the key modules. When only the SRSN module is introduced without any bootstrap alignment methods, Cv do not contribute to improving the recognition accuracy compared to when SRSN is not included. This also shows that the direct introduction of spatial representations is not suitable for FER. When only the AGM module is added, which only ensures that Cvk and Csk are aligned at the stage of REN, the accuracy of our method on the three validation database drops by an average of 1.93%. When both SRSN and LBA modules are added, the extent of the decline is attenuated. The two ablation experimental schemes differ in model design in that the former tends to realize bootstrap alignment in REN and the latter tends to realize bootstrap alignment in RFN. The experiments show that the alignment design models for the front and rear modules work, with the latter being somewhat more effective.

Influence of the K value. We evaluate the recognition performance of our method with the different values of K in Eq. (5), as shown in Fig. 4.

In particular, we can observe that our method achieves the best recognition accuracy when value of K is set to 9. When separated into too many components, the basic representations are separated too finely, weakening the representation in someone component. However, too few results in merging of components, which makes it impossible to obtain spatial information between the merging components and results in difficult to bootstrap better feature fusion.

Influence of the λ weight. As shown in Fig. 5, we can see that CS-GBSBF achieves the best performance when the value of λ is set to 0.1. When a large weight of LBA is set, the model will ignore the loss of cross-entropy, resulting in a worse classification effect. However, when the value of λ is set to 0, the lack of consideration of feature alignment in RFN leads to worse fusion.

Influence of backbone in VREN. We show our CS- GBSBF with different backbones in Table 3, and give the accuracy of CS-GBSBF as a metric for comparison. CSGBSBF with different backbones can work well. Results prove it that swinT-base [Liu et al., 2021] is able to extract Fvb that are more suitable for CS-GBSBF.

4.4 Comparisons with State-of-the-Art Methods

Table 4 shows the comparison results between the proposed CS-GBSBF method and several state-of-the-art FER methods on all databases shown in Table 1. Only a different portion of the databases in Table 1 is generally selected in the article for each comparison method, and not all comparison methods have open source code. Therefore, for each database, we try to find the state-of-the-art approach to compare with our proposed method.

As shown in Table 4, the proposed CS-GBSBF method basically improves the recognition accuracy on all databases with an average improvement of 2.48%. This is significantly improved by 2.71% and 7.17% in Oulu-CASIA and FER2013, respectively. Even on the micro-expression database, SAMM, CS-GBSBF far outperforms the compared methods on Acc. and UF1. It also improves on all other databases except RAF-DB. These comparison experiments with the stateof-the-art method clearly demonstrate our advantage in FER as well as the existence of the proposed CS-GBSBF method with high recognition generalization on different databases.

4.5 Visualization

Attention visualization. We use CAM [Zhou et al., 2016] to draw the attention heat maps based on Cv and Cs respectively, as shown in Fig. 6. We can observe that after passing through RSN, the attentional attention regions of each Cv and Cs are different but the attention regions of both correspond to each other. All the regions are distributed near the different component organs of face, which is one of the reasons why we call proposed method component separation. Similarly in exploring the attention regions of Cs on graphs, the graph is visualized and plotted, and then the regions of attention obtained by CAM are appended to the visualized graphs in the same way as images.

5 Conclusion

In this paper, we propose a novel CS-GBSBF method consisting of REN, RSN, and RFN. To address the problem that current researches focus too much on visual information and ignore other valid information, we import granular balls to represent images and extract visual and spatial information from the source of data. In our method, to address the arising problem of features alignment, we propose AGM and LBA in the stages of REN and RFN, respectively. We conducted ablation studies to demonstrate the effectiveness of our proposed modules. The extensive experimental results on multiple databases demonstrate the superiority of our method for facial expression recognition.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China (Nos. 62221005, 62476052, 62450043, 62222601), Sichuan Science and Technology Program (No. 2024NSFSC1473), and Central Guidance for Local Science and Technology Development Fund Projects (No. 2024ZYD0268).