VSGT: Variational Spatial and Gaussian Temporal Graph Models for EEG-based Emotion Recognition

1 Introduction

Emotion recognition aims to identify the emotional state of an individual as a way to study the effect of emotions on human cognition. Compared with the behavioral clues of emotions, such as body language [Coulson, 2004], voice [Zeng et al., 2007], and expression [Zhu et al., 2017], the electroencephalogram (EEG) is thought to be the outcome of synchronized neuronal activity and signal transmission that can

...

directly reflect genuine emotion, which makes EEG widely used in emotion recognition. The EEG signals are collected from multiple electrodes placed over different brain regions and exhibit various structural characteristics, including spatial correlations among brain regions within a time slice and temporal correlations between time slices, as shown in Figure 1. These spatial and temporal dependencies are specifically critical for recognizing emotional states, and thus, exploiting such dependencies in EEG-based emotion recognition has recently become mainstream [Zhou et al., 2023; Liu et al., 2024].

Despite much progress made by existing emotion recognition methods [Jia et al., 2021; Li et al., 2018], there are still two deficiencies in terms of spatial and temporal dependencies that limit the performance of emotion recognition. First, prior knowledge is rarely used to capture the spatial dependency of brain regions. The prior knowledge refers to the established physiological paradigms that can depict the dynamics and stability of fundamental connectivity patterns of brain regions [Gonzalez-Castillo et al., 2015; Braun et al., 2015]. It contributes to the deep exploitation of emotional states. Unfortunately, very few emotion recognition methods take prior knowledge into account to capture the spatial dependency.

Second, the cross temporal dependency between consecutive time slices for different brain regions is ignored. Studies have shown that emotion persistence involves the asynchronous activity in brain regions, which represents the temporal delay in the response of other brain regions to the current brain region activity [Lewis, 2005] (e.g., the correlation between vT-1 2 and vT 4 in Figure 1). Neglecting this cross temporal dependency may fail to describe the critical delayed response in brain regions for emotional states. However, for the temporal dependency between consecutive time slices, existing works [Feng et al., 2022; Li et al., 2019; Li et al., 2021b] only consider the correlations in the same brain region, ignoring those between different brain regions.

To solve the above limitations, we draw inspiration from neuroscience to better guide the utilization of spatial and temporal dependencies and propose variational spatial and Gaussian temporal (VSGT) graph models for emotion recognition. We first design a spatial and temporal EEG-to-graph (S&TE2G) block to transform the EEG data into a time-serial graph structure. To leverage the prior knowledge to facilitate the learning of emotion recognition models, we introduce the neural field theory to model the prior distribution of spatial dependency and present a variational spatial encoder (VSE) to compute a derivable posterior distribution to approximate the prior distribution, through which the spatial dependency can be captured with the variational Bayesian method. To capture the cross temporal dependency between consecutive time slices for different brain regions, we propose a Gaussian temporal encoder (GTE), in which a conditional Gaussian graph transform is used to calculate comprehensive temporal dependency between consecutive time slices. Finally, we use a graph convolutional network (GCN) to recurrently learn effective feature representations from the spatial and temporal dependencies for emotion recognition.

The contributions of the paper are summarized as follows.

• We introduce the prior knowledge i.e., neural field theory, to the spatial dependency and propose a variational spatial encoder (VSE) to capture the spatial dependency.
• We analyze the importance of cross temporal dependency for emotion recognition and propose a Gaussian temporal encoder (GTE) to capture the cross temporal dependency.
• We designed a recurrent spatial and temporal module (RST) to learn the spatial-temporal feature representations for time-serial graph structure.

2 RelatedWork

Currently, more and more researchers are devoted to analyzing EEG-based emotion recognition based on spatial or spatio-temporal dependency in graph modeling. For spatial dependency modeling, existing methods are divided into predetermined and dynamic dependencies. Predetermined dependency indicates that spatial dependency is built on certain physiological paradigms. Based on 'the decay of brain region correlations with physical distance' [Salvador et al., 2005], [Zhong et al., 2020; Xu et al., 2023] apply spatial coordinates of electrodes to calculate spatial dependency related to physical distance; Based on 'the International 10-20 EEG Electrode Standard', [Deng et al., 2021] set spatial dependency based on fixed 2D electrode connection graph; Based on 'the asymmetry of neural activity in the left and right hemispheres during emotional states' [Schmidt and Trainor, 2001], [Du et al., 2022; Ding et al., 2024] select specific cross-hemisphere electrode pairs to set spatial dependency. Dynamic dependency indicates that the spatial dependency can be updated during training. [Song et al., 2018; Zhang et al., 2019] utilize a parameter matrix to represent spatial dependency, which can be updated through the loss function; [Li et al., 2021a; Ye et al., 2022] calculates the phase difference between electrode signals as the spatial dependency; [Zeng et al., 2022; Jin et al., 2021] apply an attention mechanism that computes the adjacency matrix with model parameters.

Although predetermined dependency conforms to prior physiological paradigms, implementing prior constraints results in their inability to be continually updated, thus limiting the model's adaptability to different data. Although dynamic dependency can be continuously updated, it lacks physiological plausibility since it is not constrained by prior knowledge.

For temporal dependency modeling, graph-based methods in EEG-based emotion recognition typically combine GNN and temporal encoders. [Feng et al., 2022; Li et al., 2019; Li et al., 2021b] learn the spatial dependency within each time slice independently using GNN and then concatenated the feature representations of all time slices together to learn temporal dependencies between time slices via LSTM. Similarly, [Peng et al., 2022; Ding and Guan, 2023] utilize this structure, with the difference being the use of Transformer encoders as temporal encoders. In addition, [Jia et al., 2021; Liu et al., ] forms a series of graphs from spatial dependencies of all time slices and uses the Gate Recurrent Unit (GRU) to infer the temporal dependencies between time slices. Also, [Li et al., 2021a] averages the spatial dependencies of all time slices as the final spatial dependency of the emotional EEG.

For the aforementioned methods, despite differences in the choice of the temporal encoder, all apply a two-step process that captures spatial dependency within time slices independently and then utilizes a temporal encoder to capture temporal dependency across time slices. However, this temporal dependency ignores the correlations between different brain regions across time slices, which is a significant part of the temporal dependency of emotional EEG. As a result, the temporal dependency that fails to describe the critical delayed response of brain regions is incomplete, which in turn restricts the model performance.

3 Methodology

3.1 Overall Structure

The proposed model processes emotional EEG signals in time series and outputs a predicted emotion label. Given emotional EEG signal X ∈ RC×N with C the EEG channels and N the number of samples, the emotion recognition task is defined as learning a model F (·), which maps X into the corresponding emotion label Y = F(X).

EEG signals are derived from different brain regions and exhibit dynamic changes in connectivity patterns between brain regions over time. According to such characteristics, we first model the EEG data as time-serial graphs, then capture the spatial and temporal dependencies in EEG graphs, and finally adopt a graph convolutional network (GCN) to extract feature representations for emotion recognition.

Figure 2 shows the overall structure of our variational spatial and Gaussian temporal graph models (VSGT), which consists of recurrent spatial and temporal modules (RSTs) and a classifier. Each RST contains a spatial&temporal EEG-tograph (S&T-E2G) block, a variational spatial encoder (VSE), and a Gaussian temporal encoder (GTE).

S&T-E2G To process the EEG data in a graph space, we devise an S-E2G block and a T-E2G block to transform the EEG signals into a time-serial graph structure, in which the nodes represent the brain regions (i.e., EEG channels) and the edges represent the connections between different brain regions. The resulting EEG graphs will be further processed by VSE and GTE to characterize dependencies. See Section 3.2 for the details of S&T-E2G.

VSE The VSE aims to capture the spatial dependency of an EEG graph in the current time slice by leveraging the prior knowledge of neural field theory. The reason we use neural field theory is as follows. The neural field theory defines the brain activity produced by emotional stimuli as a superposition of traveling waves generated by neurons. The traveling wave represents the propagation of the electrical signal from one neuron to the whole neural system, and such a propagation route measures the spatial dependency between that neuron and all the neurons connected to it. So, it is reasonable to integrate neural field theory into the modeling of dependency relations between graph nodes. The details of applying neural field theory to spatial dependency are in Section 3.3.

GTE TheGTE tries to capture the temporal dependency between consecutive time slices. The temporal dependency not only contains the traditional temporal dependency in the same brain region but also contains the cross temporal dependency ignored by existing works. The cross temporal dependency is analyzed based on the response delay phenomenon in brain regions [Lewis, 2005]. See Section 3.4 for capturing the cross temporal dependency.

RST The spatial dependency and temporal dependency produced by VSE and GTE are merged and fed into a GCN to extract node embeddings. The VSE and GTE only process two consecutive time slices. To tackle the whole EEG data with T time slices, we use a recurrent structure, i.e., RST, which updates all time slices sequentially and retains the information from the previous time slice to infer the temporal and spatial dependencies of the current time slice. This ensures that all spatial dependencies can follow prior knowledge and also allows modeling the correlations of different brain regions at different time slices. The RST and the final classifier constitute the VSGT graph model, whose optimization objective is detailed in Section 3.5.

3.2 Spatial &Temporal Node-to-Graph (S&T-E2G)

The S&T-E2G block can be considered a preprocessing for follow-up dependency analysis and feature extracting. The EEG data X are first cut into T time slices {Xt |t ∈ [1, T]} (32 samples for each time slice). For the tth time slice Xt , we employ the same setup as in [Jia et al., 2021] to extract the node embedding vt i ∈ Rd , where i is the node index and d is the embedding dimension. All the vt i in a graph constitute a node set V t . Then we concatenate every two nodes and use Conv1d h(·) and Linear layer f(·) to generate edge embeddings as follows:

where h(, ·, ) denotes a Conv1d function, f denotes a linear layer, C is the node number (i.e., channel number), and EtS and EtT are the sets of spatial edge embeddings and temporal edge embeddings for the tth time slice. Note that the Conv1d and Linear layers in T-E2G and S-E2G are different.

3.3 Variational Spatial Encoder (VSE)

To better leverage the prior knowledge, the regularities of prior knowledge first need to be characterized. As discussed in Section 3.1, emotional brain activity is a superposition of traveling waves generated by innumerable neurons. According to Thermodynamic Limit, when the number of neurons tends to infinity, the coupling of an infinite number of traveling waves provides an infinite approximation to the actual spatial dependency of the brain. We define this approximation as a Binomial distribution:

where P(ei,j) represents the overall edge distribution between two nodes vi and vj, pi,j denotes the probability of edge existence within each traveling wave, and B(·) is the Binomial distribution. n →∞ corresponds to infinite neurons, and thus, the number of edges tends to infinity, which is utilized to simulate the infinite potential neuronal connections in the brain. pi,j → 0 indicates that single neural activity is physiologically slight and unobservable.

We assume the prior distribution as the distribution of the spatial dependency in the t - 1th time slice:

where pt-1 S ∈ RNe (Ne = C(C -1) is the number of edges) is computed from Et-1 S by the linear layer f.

We also assume the distribution of spatial dependency in the current time slice matches such a prior distribution. Inspired by VRNN [Chung et al., 2015], an RNN encoder is applied to estimate the parameters of the approximate posterior distribution. However, the posterior distribution of spatial dependency cannot be computed like [Chung et al., 2015]

due to the presence of infinite n. According to the De MoivreLaplace Theorem, as n tends to infinity, the binomial distribution B(n, pi,j) can be approximated by a Gaussian distribution N (npi,j , npi,j (1 - pi,j )). Based on this, we calculate a Gaussian distribution proxy to void the direct parameterization of both the infinite parameter n and the near-zero parameter pi,j .

Given a Gaussian distribution N(µS, σS 2 ), where µS ∈ RNe and σS ∈ RNe are computed from EtS and letting µS < 12 , we define the posterior Gaussian distribution as

where m denotes the mean of the Gaussian approximation of the binomial distribution and is defined as

where λ is an approximation used to prevent explosions in the calculation, and it is defined as

where ε is a very small hyperparameter (Set to 1e - 2) and ζ(·) is a softplus function.

The P(EtS ) is a Gaussian proxy to the Binomial distribution. Therefore, the P(EtS ) approximates the prior distribution while also allowing the use of the reparameterization trick [Kingma and Welling, 2013], which allows this distribution to be derived. We draw samples from such a distribution as the spatial dependency of the current time slice:

where ϵ is a random variable sampled from Standard Normal distribution. GS ∈ RNe indicates the spatial dependency of current time slice.

3.4 Gaussian Temporal Encoder (GTE)

GTE infers the temporal dependency of the current time slice. It takes temporal edge embedding EtT and spatial dependency GS of the current time slice as inputs and outputs the temporal dependency GT of the current time slice. The spatial and temporal dependencies of the current time slice could be regarded as weighting each edge of spatial dependency with a Gaussian variable. We define such Gaussian variables as the temporal dependency between time slices t and t - 1. We further assume such Gaussian variables to be conditioned on spatial dependency as:

We refer to the above operation as the conditional Gaussian graph transform. GS is the spatial dependency of time slice t from Eq. (8). N(µT , σT 2 ) is a Gaussian distribution generated from EtT . The purpose of conditional probability is that the delayed response between brain regions is based on the existence of interactions between brain regions. Therefore, we conditioned spatial dependency samples to indicate the presence of interactions between brain regions that are likely to form a delayed response. The temporal dependency of time slice t is defined as samples from such a conditional distribution:

where ϵ is a random variable sampled from Standard Normal distribution. GT ∈ RNe indicates the temporal dependency between time slices t and t - 1.

3.5 Learning of VSGT

The loss function of VSGT contains two parts: the difference between the prediction and the ground truth labels and the Kullback-Leibler Divergence (KLD) between the prior and posterior distributions in VSE. The significance of the second part is to constrain the spatial dependency distribution of the current time slice by the spatial dependency distribution of the previous time slice that conforms to prior knowledge, which further ensures the stability of all spatial dependencies, i.e., they all conform to the fundamental connectivity patterns. The loss function L is as follows:

where w is loss weight (Set to 0.1). While BCE{·} is easy to compute, the KLD part of L cannot be computed directly because the prior probability P(Et-1 S ) is a Binomial distribution B(n, pt-1 S ) with an infinite parameter n. Since minimizing the KLD part can also be regarded as minimizing the upper bound, the second part of L is defined as:

4 Experiment

Datasets We examine our method on two different datasets: DEAP [Koelstra et al., 2011] and DREAMER [Soleymani et al., 2011]. They are both EEG signal datasets tailored for human emotion analysis. Subjects will watch videos and record their emotional responses characterized by valence (high/low) and arousal (high/low). They evaluate the valence and arousal on a scale from 1 to 9. We adopt the general EEG-based emotion recognition paradigm in which valence and arousal scores are divided into two categories bounded by 5. Separate experiments were conducted in each subject and averaged as the final result of the dataset. We present the result on value and arousal separately in our experiment results.

Training Paradigm We adopt the k-fold cross-validation paradigm to divide the training and testing sets. For each subject, the data is divided into 10 folds, where one fold is the testing set, and the remaining are used to train. The data used for training is divided into 8 : 1, where one fold is the validation set, and the remaining is the training set.

Cropping Strategy To expand the data volume, we used a cropped strategy ([Schirrmeister et al., 2017]), which crops the data of each trail into 4s non-overlapping segments. The cropping is done strictly after splitting the training, validation, and testing sets, which prevents data leakage. We make all baselines follow the same cropping strategy to prevent abnormally high accuracy and F1 scores.

Baseline and Hyperparameters We compared VSGT with three types of baselines. One category is the non-graph methods; another is the graph-based methods; and the last one is the methods combining GNN and temporal encoders. By comparing with these baselines, we hope to demonstrate that the VSGT can make full use of spatio-temporal dependencies to improve the accuracy of EEG-based emotion recognition. We set up a shared environment for all baselines to ensure consistency except for hyperparameters.

4.1 Comparison with Prior Art

The accuracy and F1 scores of the test results are shown in Table 1. Compared to non-graph methods, VSGT achieves a significant improvement in performance on two datasets. MM-ResLSTM [Ma et al., 2019] and ACRNN [Tao et al., 2020] utilize a temporal encoder to capture the temporal dependency of emotional EEG. TSception [Ding et al., 2022] and SST-EmotionNet [Jia et al., 2020] both apply convolution in the spatial and temporal dimensions of the signal to learn spatial and temporal dependencies. The combination of spatio-temporal dependencies allowed TSception and SSTEmotionNet to achieve better results. But essentially, the above baselines' temporal dependency inference process is still performed independently, which ignores the correlations between different brain regions across time slices and thus performs worse than VSGT.

Compared to graph-based methods, VSGT still demonstrated superior performance. RGNN [Zhong et al., 2020] and LGGNet [Ding et al., 2023] both introduce specific prior knowledge as the spatial dependency. The prior knowledge used by RGNN to define spatial dependency is that correlations between brain regions show regular decay with physical distance. Therefore, RGNN calculates spatial distances from the 3D coordinates of the electrodes and uses this as the spatial dependency. LGGNet relies on prior knowledge of existing correlations between functional brain regions and emotions. Connections between electrodes were predefined based on specific brain region divisions. However, since the prior knowledge is fixed, the spatial dependency cannot be adjusted during training. Instead of using prior knowledge, both DGCNN [Song et al., 2018] and MTGNN [Wu et al., 2020] parameterize the spatial dependency as part of the model, which is updated by loss. All of the above methods transform the time series signals into frequency-domain features and, therefore, do not include temporal dependency. Also, not implementing the use of prior knowledge to capture spatial dependency further limits their performance.

Compared to methods combining GNN and temporal encoders, VSGT benefits from better results due to more comprehensive spatial and temporal dependencies. HetEmotionNet [Jia et al., 2021], ASTG-LSTM [Li et al., 2021b], STGCLSTM [Feng et al., 2022] and EmoGT [Jiang et al., 2023]

utilize the same spatial and temporal dependency inference process: model the spatial dependency within each time slice using GNNs, and apply a temporal encoder to infer the temporal dependency of all time slices. The difference is that HetEmotionNet uses Gated Recurrent Units, ASTG-LSTM and ST-GCLSTM use LSTM, and EmoGT uses a transformer encoder. Due to the introduction of temporal dependency and the construction of more comprehensive spatial dependency via GNNs, the above approach achieves better results than the other two types of approaches. However, the spatial and temporal dependencies inference approach they used was unable to establish correlations between different brain regions across time slices, hence the better results achieved by VSGT.

4.2 Interpreting Spatial Dependency

We further verified the interpretability of the spatial dependency inferred by VSGT on DEAP in Figure 3. For readability reasons, we only show part of the dependencies between

21 electrodes. The spatial dependency can be summarized as follows: For positive emotions, electrodes F3, F4, Fp1, and Fp2 showed high correlations at different time slices; for negative emotions, electrodes C3 and C4 showed high correlations at different time slices. This phenomenon is consistent with [Min et al., 2022]'s conclusion that the brain's frontal lobe (Fp1, Fp2) and frontal cortex (F3, F4) show a high correlation under positive emotion; under negative emotion, the central sulcus (C3, C4) showed a high correlation. Therefore, the spatial dependency maintains stability (i.e., the high correlation region is the same for different time stamps) and also maintains dynamics (the high correlation shows a certain degree of fluctuation).

4.3 Interpreting Temporal Dependency

We visualized several sets of signals from different channels with high correlation in continuous time slices in Figure 4. In Figure 4 (a), Signal 2 has an amplitude decrease at the time slice t - 2 and an amplitude increase at the time slice t, while signal 1 has an amplitude decrease only at the time slice t-1 and an amplitude increase at the time slice t + 1. This suggests that the VSGT noticed that there was a significant delay in the response time of Signal 1, even though both Signal 1 and Signal 2 appeared similarly trough. In Figure 4 (b), Signal 4 appears to have successive peaks at the time slices t-1 and t + 1, whereas similar successive peaks for signal 3 occur at the time slices t and t - 2. This similarly suggests that there is a response delay in the brain regions corresponding to these two signals. In Figure 4 (c), the similar troughs of Signal 1 and Signal 2 begin at the same time slice while the signal 2 reaches its lowest point one time slice later than the signal 1. In Figure 4 (4), Signal 3 and Signal 4 all show a significant amplitude decrease, but the decrease of Signal 4 starts later and lasts longer. We suggest that these correlations between different brain regions across time slices contain important temporal dependency information. For example, from the physiological perspective, this delay responds to asynchronous activity between brain regions; from the signal processing perspective, the channel in which the first response occurs may be closer to the center of activity in the brain. Thus, the time dependency captured by the VSGT is physiologically plausible.

4.4 Ablation Study

We performed ablation studies to assess the impact of VSE and GTE and their core components. Figure 5 illustrates the results for the four cases. 1) Without VSE: Removing the VSE brings the least impact on the model because the temporal dependency inferred by the GTE is built on two neighboring time slices, which also contain a certain degree of spatial dependency information. 2) Without GTE: Removing GTE has the greatest impact on the model because when there is no GTE, the KLD part of Loss makes the spatial dependency of each time slice converge to the same, and this same spatial dependency is directly used to update the node embedding. 3) Without Condition: Not using the temporal dependency distribution as the conditional probability distribution of the spatial dependency results in the temporal dependency distribution behaving randomly when some samples of the spatial dependency converge to zero. 4) Without KLD: Removing the KLD part loss makes the VSE a simple linear module, and the spatial dependency of the current time slice can neither be correlated with the previous time slice nor be constrained by prior knowledge, so the result is relatively close to the direct removal of the VSE.

5 Conclusion

We propose the VSGT for EEG-based emotion recognition. Through VSE, VSGT realizes spatial dependency that is based on prior knowledge and can be updated. With GTE, VSGT captures the correlation of different brain regions across time slices in the temporal dependency of emotional EEG, which has been neglected in the past. RSTs integrate spatial-temporal dependencies with a recurrent structure and update the representation of each time slice. The experimental results show that the comprehensive spatial-temporal dependencies of EEG can be learned and utilized to improve the accuracy of EEG-based emotion recognition. In addition, interpretability experiments showed that the spatial and temporal dependencies inferred by the VSGT are consistent with existing physiological phenomena, which further explains the underlying working mechanism of the model.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 62372334.

Contribution Statement

†Co-first Authors, ∗Corresponding Authors