Multi-level Disentangling Network for Cross-Subject Emotion Recognition Based on Multimodal Physiological Signals

1 Introduction

Emotion recognition plays a pivotal role in affective computing [Cowie et al., 2001]. In recent years, researchers have typically employed both non-physiological signals and physiological signals for emotion recognition [Shen et al., 2019]. Non-physiological signals such as text, video, and audio are easily influenced by subjective factors and can be easily masked [Deng et al., 2018]. Hence, it is not guaranteed to reflect human emotional states accurately. In contrast, some crucial physiological signals such as electroencephalogram (EEG), electromyogram (EMG), and electrooculogram (EOG) can objectively represent the true emotional state of the human body. As a result, emotion recognition based on physiological signals has gradually become a hot research topic [Ning et al., 2023].

∗Corresponding Author

Compared to unimodal physiological signals, multimodal physiological signals can represent emotional states comprehensively. Therefore, some emotion recognition methods based on multimodal physiological signals have achieved state-of-the-art performance [Jia et al., 2021; Zhang et al., 2020a; Abdullah et al., 2021; Zhang et al., 2022]. However, there are still two major challenges for emotion recognition based on multimodal physiological signals:

1) How to model the consistency and heterogeneity of multimodal physiological signals simultaneously. The consistency and heterogeneity are two important characteristics of multimodal physiological signals [Zhang et al., 2020a; Khan et al., 2023]. Specifically, the consistency of multimodal physiological signals refers to the interrelatedness of different physiological signals for the same physiological activity. In emotion recognition, different physiological signals can reflect the same emotional state, providing comprehensive information. As shown in Figure 1, both EEG and EOG reflect a happy emotional state. The heterogeneity of multimodal physiological signals refers to distinct features between different physiological signals [Jia et al., 2023]. As shown in Figure 1, EEG and EOG exhibit notable differences in temporal and spectral domain distributions. However, most existing methods model either the consistency or heterogeneity of multimodal physiological signals individually. To model consistency, joint models project different physiological signals into a common subspace for feature extraction [Lu et al., 2015; Yin et al., 2017; Xu et al., 2021; Jia et al., 2021], but ignore the heterogeneity between different physiological signals. To model heterogeneity, coordination models typically use different feature extractors for each physiological signal to capture their distinct features [Qiu et al., 2018; Ma et al., 2019b; Zhang et al., 2020b; Jia et al., 2022b; Lai, 2004], yet overlook consistency. Thus, how to model the consistency and heterogeneity of multimodal physiological signals simultaneously still remains a challenge.

2) How to model individual differences across subjects in emotion recognition. Due to structural and functional differences between subjects, physiological signals are different even though the subjects are in the same emotion state [Liu et al., 2024; Zhang et al., 2018]. This leads to poor performance in cross-subject emotion recognition [Fdez et al., 2021]. Some researchers attempt to use domain adaptation to improve model performance in cross-subject emotion recognition [Zhao et al., 2021; Li et al., 2019a]. Although they can achieve excellent performance, Domain adaptation requires collecting extensive data from new subjects to customize the model in practical application, making the practical application of emotion recognition models inconvenient. Therefore, how to model individual differences across subjects in emotion recognition still remains a challenge.

To address these challenges, we propose a Multi-level Disentangling Network named MDNet for cross-subject emotion recognition based on multimodal physiological signals. MDNet consists of a modality-level disentangling module and a subject-level disentangling module. The modality-level disentangling module includes a modality-invariant encoder and several modality-specific encoders. The subject-level disentangling module comprises a subject-shared encoder and several subject-private encoders.

Overall, the main contributions of our work are summarized as follows:

• We propose a modality-level disentangling module, which captures modality-invariant features and modality-specific features. Both the consistency and heterogeneity of multimodal physiological signals are integrated into a unified framework.
• We develop a subject-level disentangling module, which separates both subject-shared features and subjectprivate features, in order to address the individual differences across subjects.
• Experiments on two multimodal emotion datasets show that MDNet achieves state-of-the-art performance.

2 RelatedWorks

2.1 Multimodal Emotion Recognition

Multimodal emotion recognition models can mainly be classified into joint models and coordination models. Joint models extract features of multimodal physiological signals in the same subspace to model the consistency of multimodal signals [Xu et al., 2021; Jia et al., 2021; Lu et al., 2015; Liu et al., 2023; Yin et al., 2017]. Xu et al. [2021] treat multimodal physiological signals as a multi-dimensional tensor. They integrate the multi-scale characteristics by fusing multi-core information. Jia et al. [2021] propose a twostream heterogeneous graph recurrent neural network, which achieves multimodal feature fusion of temporal, spectral, and spatial features. Coordination models, on the other hand, extract features of multimodal physiological signals in separate subspaces and apply certain correlation constraints to these features to model the heterogeneity of multimodal signals [Qiu et al., 2018; Jia et al., 2022a; Zhang et al., 2020b]. Qiu et al. [2018] utilize deep canonical correlation analysis (DCCA) for emotion recognition, which can learn separate representations for each modality in a non-linear way. Ma et al. [2019b] design a multimodal residual LSTM network for emotion recognition, projecting different modalities into separate LSTM branches to extract multimodal features.

Although joint models and coordination models can achieve high accuracy, they do not simultaneously model the consistency and heterogeneity of multimodal physiological signals. Our model combines the advantages of both joint models and coordination models, modeling the two characteristics within the same framework.

2.2 Cross-Subject Emotion Recognition

Individual differences exist in inter-subject physiological signals [Jia et al., 2022b]. The variations of physiological signals from different subjects limits the model's performance for cross-subject emotion recognition. To address this challenge, researchers have proposed a series of methods to enhance emotion recognition models' generalizability [Ghifary et al., 2016; Ma et al., 2019a; Ganin et al., 2016; Li et al., 2018]. Li et al. [2019b] use source selection and style transformation mapping to reduce the domain differences between target and source, facilitating the acquisition of common features in EEG. Zhao et al. [2021] develop a plug-and-play domain adaptation method. This method customizes the model by inputting EEG from target subjects to enhance the model's specificity, which makes emotion recognition more generalizable and practicable as well. Li et al. [2019a] improve the model's generalizability by minimizing the classification error on the source while making the source and the target similar in latent representations. Li et al. [2021] propose a Transferable Attention Neural Network (TANN), which distinguishes the contribution of different samples for emotion recognition and uses local and global attention mechanisms to highlight the discriminative EEG data adaptively.

The methods above can effectively deal with individual differences across subjects and improve the accuracy of models in cross-subject emotion recognition. However, these methods only consider the individual differences in unimodal EEG and do not take a further concern of multimodal signals for cross-subject emotion recognition.

3 Multi-level Disentangled Network

Our goal is to achieve high performance in cross-subject emotion recognition by modeling key features from multimodal data. As shown in Figure 2, MDNet consists of a modalitylevel disentangling module and a subject-level disentangling module. Specifically, the modality-level disentangling module captures modality-invariant features and modalityspecific features from multimodal physiological signals. A self-attention mechanism is then used to fuse these features to obtain subject-level representation, reducing the information redundancy when fusing multimodal features. Afterwards, subject-level disentangling module separates subject-shared features and subject-private features from the subject-level representation. In order to deal with the individual differences across subjects, only subject-shared features are used for classification, enhancing the accuracy of emotion recognition.

3.1 Modality-level DisentanglingModule

To model key features from multimodal data, modalityinvariant features and modality-specific features are captured by projecting each modality into modality-invariant subspace and modality-specific subspace, respectively. In the modality-invariant subspace, modality-invariant features learn the multimodal consistency. In the modality-specific subspace, modality-specific features learn the multimodal heterogeneity. Afterwards, modality-invariant features and modality-specific features for each modality are obtained. A multi-head self-attention mechanism is applied to these features, which allows each feature to induce potential information from other features that are synergistic for emotion recognition.

Specifically, for each modality, a modality encoder is used to initially extract the modality features. The encoding process can be formally expressed as:

where Xi ∈ RCi×Li represents the input of the i-th modality, Ci denotes the number of channels in the i-th modality, Li denotes the number of sampling points in each channel of the ith modality, and n represents the number of input modalities. Then, we obtain the initial features ui(i = 1, 2, · · · , n) ∈ Rd extracted from all modalities, where d represents the dimension of each feature.

The modality-level disentangling module consists of a modality-invariant encoder and n modality-specific encoders. The modality-invariant encoder re-encodes the initial features ui(i = 1, 2, · · · , n) of the modalities, projecting them to the modality-invariant subspace. The encoding process of the modality-invariant encoder can be formally expressed as:

where Et represents the modality-invariant encoder composed of fully connected layers, θt denotes the parameters of the modality-invariant encoder, and u˜i represents the modalityinvariant features of i-th modality. For all modalities, the modality-invariant encoder Et shares the parameters θt, enabling the re-encoded features to learn the multimodal consistency. Hence, we obtain the modality-invariant features u˜i(i = 1, 2, · · · , n) ∈ Rd extracted from the initial features of all modalities.

The modality-invariant features of different modalities in the modality-invariant subspace have to be as similar as possible. We use Central Moment Discrepancy (CMD) to measure the similarity between modality-invariant features. The CMD decreases as the distribution of modality-invariant features become more similar. The modality similarity loss Lmsim is defined as:

For i-th modality, the i-th modality-specific encoder reencodes the initial features ui, projecting them to a modalityspecific subspace. The encoding process of the modalityspecific encoder can be formally expressed as:

where Ec represents the modality-specific encoder composed of fully connected layers, θi denotes the parameters of the modality-specific encoder, and uˆi represents the modalityspecific features of i-th modality. For each modality, the modality-specific encoder E i c has different parameters θi , enabling the re-encoded features to learn the individual differences across subjects. Consequently, we obtain the modalityspecific features uˆi(i = 1, 2, · · · , n) ∈ Rd extracted from the initial features of all modalities.

Modality-invariant features and modality-specific features represent different aspects of a same modality. Orthogonality constraint is used to maximize the difference between modality-invariant features and modality-specific features. Similarly, the differences between modality-specific features of different modalities are maximized by the same orthogonality constraint. The modality difference loss Lmdiff is defined as:

where || · ||2F represents the squared Frobenius norm.

During the re-encoding process, some information contained in the initial features is inevitably lost. Mean squared error is used to minimize the distortion of the re-encoded features. To measure the distortion of re-encoded features, we use a decoder composed of fully connected layers to reconstruct the initial features u¯i = D(u˜i + uˆi, θd ). The modality reconstruction loss Lmrecon is defined as:

where || · ||22 represents the squared L2-norm.

After obtaining modality-invariant features and modalityspecific features of all modalities, we concatenate these features to obtain the modality-level representation H = (u˜1||uˆ1||u˜2||uˆ2 · · · u˜n||uˆn) ∈ R2n×d , where || denotes concatenation. To induce interaction between features, we introduce the attention mechanism [Vaswani et al., 2017], which can be formally expressed as:

where Q, K, V stand for query, key, and value, respectively, dk is the dimension of the features. Multi-head attention consists of multiple parallel attention mechanisms. The headi is defined as:

where W Q i ,W K i ,W V i are the parameters of the i th attention mechanism.

In order to obtain the subject-level representation h, we apply the multi-head self-attention mechanism. Formally, the subject-level representation is computed as:

where θh = {W Q,W K ,W V ,W O }, the definition of headi is as given above, and || denotes concatenation.

3.2 Subject-level DisentanglingModule

After obtaining the subject-level representation, we use the subject-level disentangling module to separate subject-shared features and subject-private features. Subject-shared features represent the similar expression of different subjects for the same emotion. On the other hand, subject-private features represent the personalized expression of different subjects for the same emotion.

The subject-level disentangling module consists of a subject-shared encoder and m subject-private encoders, where m is the subject's number in the training data. The subject-shared encoder re-encodes the subject-level representation to capture the subject-shared features, which can be formally expressed as:

where Es represents the subject-shared encoder composed of fully connected layers, θs denotes the parameters of the subject-shared encoder, and h˜ represents the subject-shared features. For all subjects, the subject-shared encoder Es shares parameters θs, enabling the re-encoded features to learn the similarities among different subjects. As a result, we obtain the subject-shared features h˜ ∈ Rd extracted from the subject-level representation.

For each subject, a subject-private encoder re-encodes their subject-level representation to capture the subject-private features, which can be formally expressed as:

where E ip represents the subject-private encoder composed of fully connected layers, θip denotes the parameters of the i-th subject-private encoder, and hˆ represents the subject-private features. For each subject, the subject-private encoder E ip has different parameters θip , enabling the re-encoded features to learn the uniqueness of different subjects. As a result, we obtain the subject-private features hˆ ∈ Rd extracted from the subject-level representation.

Subject-shared features and subject-private features reflect different aspects of human emotional characteristics. In order to capture different aspects of the subject-level representation, an orthogonality constraint is applied to maximize the differences between subject-shared features and subjectprivate features. The subject difference loss Lsdiff is defined as:

where || · ||2F represents the squared Frobenius norm.

We impose constraint using mean squared error which can minimize the distortion of the re-encoded features. To measure the distortion of features after the separation process, we use a decoder composed of fully connected layers to reconstruct the subject-level representation h¯ = D(h˜ +hˆ, θd ). The individual reconstruction loss Lsrecon is defined as:

where || · ||22 represents the squared L2-norm, d is the dimension of h.

Finally, subject-shared features h˜ are fed into a classifier Cs. The result can be formally computed as:

where yˆ is the emotion classification result.

The cross-entropy is used as the task loss Ltask, which is defined as:

where yi represents the label of the task, n is the number of samples in a batch.

3.3 Learning Loss

A joint loss function is employed to constrain the model, formally, the entire loss function can be represented as:

where α, β, γ, δ, ϵ are the weights of the loss terms. The task loss Ltask is used to ensure accurate classification. The Modality similarity loss Lmsim ensures that the modalityinvariant features remain similar. The modality difference loss Lmdiff guarantees the distinction between modalityspecific features of different modalities and also between modality-specific features and modality-invariant features within the same modality. The modality reconstruction loss Lmrecon minimizes distortion of modality-specific features during the re-encoding process. The subject difference loss Lsdiff ensures a clear separation between subject-shared features and subject-private features. The subject reconstruction loss Lsrecon reduces feature distortion of subject-shared features and subject-private features during the separations.

4 Experiments

4.1 Datasets

We evaluate our MDNet on two public multimodal datasets: DEAP [Koelstra et al., 2011] and MAHNOB-HCI [Soleymani et al., 2011].

The DEAP dataset is a collection comprising 40 music videos, each with a duration of one minute. The dataset is created based on the responses of 32 subjects who conducted online self-assessments. During the experiment, the 32 subjects watch the selected videos while their Electroencephalogram (EEG) is recorded using a BioSemi EEG cap with 32 channels, conforming to the international 10-20 system standards. Additionally, 8 channels are dedicated to collecting peripheral physiological signals (PPS), including respiration rate, Electromyogram (EMG), and Electrooculogram (EOG). Each subject rates the videos based on arousal, valence, liking, dominance, and familiarity. In the preprocessing stage, all signals are downsampled to 128 Hz and EEG is passed through a bandpass filter from 4 Hz to 45 Hz. Electrooculogram (EOG) artifacts are removed. For the DEAP dataset, the arousal and valence ratings range from 1 to 9. Thus, ratings ≥ 5 are labeled as positive for high arousal or valence, and those < 5 as negative for low arousal or valence.

The MAHNOB-HCI dataset includes data from 30 subjects, each watching 20 video clips while recording physiological data for 20 trials. The length of these video clips ranges from 34.9 seconds to 117 seconds (average 81.4 seconds, standard deviation 22.5 seconds). Subjects provide subjective feedback using a score range from 1 to 9. This dataset includes EEG signals from 32 channels and 6 channels of peripheral physiological signals (PPS). Their EEG data are recorded using the Biosemi Active II system with 32 Ag/AgCl electrodes at a sampling rate of 512Hz. The peripheral physiological signals include Electrocardiogram (ECG), Galvanic Skin Response (GSR), respiration, and body temperature. We downsample all signals to 128Hz. In model evaluation, a fixed 5-point threshold is used to discretize the subjective feedback into binaries for two emotional dimensions, valence and arousal, where values ≥ 5 indicate positive, and < 5 as negative. However, data from subjects 12, 15, and 26 are missing, and thus excluded. Therefore, data from 27 out of the 30 subjects are used, consistent with existing models like TSecption [Ding et al., 2022] and HetEmotionNet [Jia et al., 2021].

4.2 Experiment Settings and Implementation

Multiple modalities are employed across two datasets: EEG, EOG, and EMG for the DEAP dataset, while EEG, ECG, and GSR for the MAHNOB-HCI dataset. To avoid data leakage during the evaluation process, we adopt a leave-one-subjectout cross-validation protocol on both datasets. Specifically, the training and test data are from different subjects, ensuring no data leakage. For example, in DEAP dataset, we use data from 31 subjects for training and the remaining subject's data for testing, repeating the validation process until each subject's data has been used as test data once. The entire experiment is conducted with 32 validations, and the average of all obtained test results is calculated as the final cross-validation performance.

To ensure a fair comparison, we apply the same treatment to our MDNet and all baseline models. Specifically, we set a 6s window with a 1s overlap. Considering the potential imbalance in label distribution in the original datasets, we resample and balance the training set labels. This helps avoid biases in the model during training and enhances its generalization ability, preventing imbalances in predicted labels.

We implement our MDNet based on the Pytorch framework. Our model is trained by Adam optimizer with a learning rate lr = 0.0003. The batch size is 64. The weights are α=0.002, β=0.1, γ=0.04, δ=0.5, and ϵ=0.05.

4.3 Baseline Methods

We compare our MDNet with eight state-of-the-art models, including SVM [Chatterjee and Bandyopadhyay, 2016], DGCNN [Song et al., 2018], EEGNet [Lawhern et al., 2018], ACRNN [Tao et al., 2020], SST-EmotionNet [Jia et al.,

2020], HetEmotionNet [Jia et al., 2021], TSception [Ding et al., 2022], LGGNet [Ding et al., 2023].

4.4 Experiment Analysis

To validate the effectiveness of our MDNet, we compare it with eight baseline methods on DEAP dataset and MAHNOB-HCI dataset, as shown in Table 1. Specifically, traditional models like EEGNet primarily contribute to extracting effective features from EEG signals to improve the performance of emotion recognition. However, these models neglect the significance of multimodal signals and do not utilize the fusion of multiple modalities to improve performance. In contrast, HetEmotionNet effectively extracts discriminative features from multimodal signals. It can model the heterogeneous information of different modalities, making it perform better than EEG-based models. TSception is able to extract discriminative features from multiple modalities, which in turn improves model performance. However, these models overlook the individual differences across subjects in multimodal signals, which often reduces the classification accuracy. In comparison, our MDNet employs disentangled representation learning to utilize the consistency and heterogeneity of multimodal physiological signals and reduce the impact of individual differences across subjects. As a result, our MDNet yields the highest accuracy in cross-subject emotion recognition tasks, and outperforms all the baseline models on both datasets.

4.5 Ablation Studies

To validate the effectiveness of our model, we conduct ablation experiments on DEAP dataset. These experiments are categorized into three types:

1. validating the effectiveness of modality-level disentangling module and subject-level disentangling module;
2. assessing the effectiveness of multimodal fusion;
3. examining the effectiveness of the loss terms.

Ablation on disentangling modules: To validate the effectiveness of the disentangling modules, we conduct ablation experiments on modality-level disentangling module and subject-level disentangling module, as shown in Figure 3. The results indicate that the classification accuracy for valence and arousal significantly decreases in the absence of the modality-level disentangling module. Similarly, the accuracy drops without the subject-level disentangling module, particularly in classifying valence. This demonstrates that both disentangling modules improve the performance of our MDNet in cross-subject emotion recognition.

Ablation on multimodal fusion: To verify the superiority of multimodal fusion, we conduct ablation experiments by removing each modality (EEG, EMG, and EOG) individually from the original setting that uses three modalities. Figure 4 shows that removing any modality results in a decrease in classification accuracy. The removal of EEG signals has the most pronounced effect on accuracy reduction. This aligns with traditional cognition that EEG has the primary contribution to emotion recognition [Zhang et al., 2020a].

It is also observed that using unimodal EEG yields better results than unimodal EOG, and using unimodal EOG performs better than unimodal EMG. This further validates the importance of EEG. The combination of multimodal data achieves better performance, which proves multimodal fusion has critical importance for emotion recognition.

Ablation on loss terms: To validate the effectiveness of the loss terms, we conduct ablation experiments, designing

five variants:

• L1 = Ltotal -Lsrecon
• L2 = Ltotal -Lsdiff
• L3 = Ltotal -Lmrecon
• L4 = Ltotal -Lmsim
• L5 = Ltotal -Lmdiff

To assess the importance of each loss term, we conduct ablation experiments by removing each loss term individually and retraining the model. Figure 5 indicates that the model performance is optimal when all loss terms are involved (Ltotal). Removing any loss term leads to a decrease in accuracy. Observations on L4 and L5 reveal that the model is most sensitive to Lmdiff and Lmsim, which validate their importance in separating modality-specific features and modalityinvariant features. Similarly, removing Lsdiff and Lssim also leads to a drop in classification accuracy, which verifies the vital role of Lsdiff and Lssim in separating subject-shared features and subject-private features. Overall, these results prove the importance of loss terms in maintaining and improving model performance and their necessity in the model optimization process.

4.6 Visualizations of DisentanglingModules

Validation of modality-level disentangling To explore the role of the modality-level disentangling module, we visualize the outputs of modality-invariant and modality-specific encoder. The t-SNE dimensionality reduction technique is utilized to transform high-dimensional data into lowerdimensional data. Additionally, a scatter plot visualization is applied to effectively present the outputs of the modalityinvariant encoder and the modality-specific encoder which are shown in Figure 6a. In this figure, different colors represent different physiological signal modalities. The greyshaded area covers the outputs of the modality-invariant encoder, while the outer area represents the outputs of the modality-specific encoder. This visualization reveals a distinction between the modality invariant representations and specific representations in the encoded data. This indicates that the modality-level disentangling module is effective in

• (a) Visualization of the outputs of modality-invariant and modality-specific encoder for the 32nd subject (test subject).
• (b) Visualization of the outputs of subject-shared encoder and subject-private encoders.

Figure 6: Visualizations of disentangling modules on DEAP dataset.

separating modality-invariant features from modality-specific features and in modeling the consistency and heterogeneity of multimodal data.

Validation of subject-level disentangling To validate the effectiveness of the subject-level disentangling module, we use the t-SNE dimensionality reduction and scatter plot visualization. Figure 6b shows the output feature distribution of the subject-shared encoder and subject-private encoders: the densely packed dots in the central area represent the shared emotional components, while the scattered dots around the periphery, separated from the central area, represent subject private components. Here, red triangles indicate data points from the test subject. This result demonstrates that the encoded outputs provide distinguishable shared and private features. The subject-shared encoder successfully extracts subject-shared features that are highly relevant to emotions, while the subject-private encoders effectively separate subject-private features.

5 Conclusion

In this paper, we propose a multi-level disentangling network for cross-subject emotion recognition based on multimodal physiological signals. To the best of our knowledge, it is the first to apply disentangled representation learning to simultaneously model consistency and heterogeneity in multimodal physiological signals, as well as the individual differences across subjects. The modality-level disentangling module captures modality-invariant features and modality-specific features to model the consistency and heterogeneity of multimodal physiological signals. The subjectlevel disentangling module separates subject-shared features and subject-private features, to model individual differences across subjects. Experimental results show that our model achieves state-of-the-art performance. In addition, ablation studies validate the effectiveness of the disentangling modules. To sum up, we propose a general framework, and we plan to further explore the application of MDNet in sleep stage classification and depression detection in the future.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (Grant No. 62306317) and STI2030-Major Projects (Grant No. 2021ZD0200200) and Postdoctoral Fellowship Program of CPSF (Grant No. GZC20232992) and China Postdoctoral Science Foundation (Grant No. 2023M733738).

Contribution Statement

Fengming Zhao, Yuzhe Guo, and Hairong Chen have equal contributions to this paper.