Online ECG Emotion Recognition for Unknown Subjects via Hypergraph-Based Transfer Learning

1 Introduction

Emotion recognition (ER) aims at recognizing emotional states of the subjects from their responses to improve user experience, thus has wide range of applications including entertainment and health care [Du et al., 2020]. With the accessibility of a massive number of physiological signals, emotion analysis using these signals is attracting increasing attention, among which electrocardiogram (ECG) plays a vital role since it allows to measure signals in the human body that are directly related to emotional states and it is suitable for wearable care [Hsu et al., 2017]. Most of the existing ECG-based ER methods focus on recognition accuracy with the offline analysis of public or specific datasets. They generally rely on a universal classifier trained on the data of known subjects to recognize the emotion states of unknown subjects. Unfortunately, since the ECG signals may diverge among different individuals and change over time, these subject-independent models inevitably suffer from performance deterioration.

To address this problem, cross-subject ER [He et al., 2021] and cross-session ER [Li and Li, 2014] have been proposed to alleviate the influence of individual differences across subjects and the nonstationary nature of ECG across-session respectively using domain adaptation techniques. These methods have a premise that the unlabeled ECG data of unknown target subjects are accessible in training phase, and leverage these unlabeled data for finetuning the recognition model trained on the labeled data of source subjects in order to achieve precise ER. As shown in Figure 1, traditional classifier cannot deal with the individual differences, domain adaptation technology needs all target data to reduce the differences between source data and target data. However, this premise does not hold in some practical scenarios where the ECG data of unknown target subjects arrive one by one instead of being acquired at a time. In this case, the data of target subjects become inaccessible in training phase, which limits existing cross-subject ER models from producing accurate emotion states predictions on target subjects.

In this paper, we propose a novel hypergraph-based online transfer learning method (HOTL) to tackle the online crosssubject emotion recognition. The proposed method is capable of modeling the high-order relationships among ECG signal via hypergraph learning, and adapting online data by updating the model with a hypergraph-based online transfer strategy. Specifically, in model training phase, hypergraph embedding is adopted to preserve the structural information of data by constructing hypergraphs which represent high-order relationships among ECG signals from different views including time domain, frequency domain, and deep features. Consequently, the constructed multi-view hypergraph which models the relationships regarding both different local views and global view can be more effectively generalized to the unknown target subjects. In online recognition phase, a hypergraph-based online transfer learning method is proposed to transfer knowledge from the hypergraph model trained on source data to unknown target data by adding a hyperedge which connects a newly coming sample with the current hypergraph, as shown in Figure 1. In this way, the hypergraph model can personalize the emotion recognition for a target individual by constantly adapting online data.

The major contributions of this work are as follows:

• We propose a hypergraph-based online transfer learning (HOTL) method to address the challenging online emotion recognition problem in which the data of unknown subject arrive one by one instead of being acquired in training phase. Unlike prevailing works which need target data of unknown subjects in training phase to adapt either retrain a new model by re-adapting the incoming data in a batch manner, our model can be easily updated to adapt ECG data changes without re-training and readapting when unknown subjects are encountered or individual states change over time.
• The hypergraph structure of proposed method can not only enhance the generalization ability of the model by revealing the high-order relationships among data which conduces to recognize unknown subjects, but also be easily adapted by adding a hyperedge which connects a newly coming sample with the current hypergraph to alleviate the influence of individual differences across subjects and nonstationary nature of ECG. Thus, our method can adapt the unknown target data well in the constantly varying online scenarios.
• Extensive experiments on the Amigos dataset demonstrate the validity and performance of our HOTL method in online emotion recognition.

2 RelatedWork

Physiological signal has been proven to be a reliable source of information for emotion recognition (ER) systems [Correa et al., 2018]. Automated physiological signal analysis can identify the affective states of users such as happiness, sadness, and stress, among others. In this paper, we mainly focus on cross-subject emotion recognition based on online transfer learning methods. To understand better, we compare our HOTL and related work in Table 1.

2.1 Cross-subject Emotion Recognition

Individual differences are unavoidable in ER and make it difficult to acquire a general model that can be suitable for all subjects [Kim and Andr´e, 2008]. The individual differences (such as personality) may cause a physiological signal discrepancy between source and target subjects, which may hinder the generalization of ER models for new subjects. The conventional method is to develop a subject-dependent model which trains a new model for a new subject using labeled data [Agrafioti et al., 2011], yet labeled data is costly to collect.

Recently, some ER methods proposed to address intersubject discrepancy by unsupervised domain adaptation (UDA), which personalized a general ER model for new subjects in an unsupervised way with knowledge transfer from source subjects [Zheng and Lu, 2016; Lan et al., 2018; He et al., 2021]. For example, [Zheng and Lu, 2016] suggested to exploit transfer component analysis (TCA) [Pan et al., 2010] to learn a shared subspace where the difference between subjects is reduced. In this way, only unlabeled data are required for target subjects. [He et al., 2021] proposed an online cross-subject ER approach from ECG signals via unsupervised domain adaptation and online domain adaptation. These methods all need data of unknown subjects in training process where target data are collected in advance. However, existing methods will fail in some practical scenarios where the data of target subjects cannot be acquired in advance but arrives one by one in an online manner. Thus, existing methods may get a performance degradation since the target data is inaccessible in training phase.

2.2 Online Transfer Learning

Online learning has been extensively studied in the machine learning community [Pan and Yang, 2009; Wu et al., 2017b; Wu et al., 2019; Zhou et al., 2020]. Perceptron [Crammer et al., 2006] simply updates a linear classifier when a new instance is classified incorrectly. [ZHAO and HOI, 2010]

first combined transfer learning with online learning, and proposed a framework online transfer learning (OTL). [Zhao et al., 2014] transferred knowledge from a single source domain to deal with an online classification task in the target domain. Using an ensemble method, [Wu et al., 2017a] exploited knowledge from multiple source domains to promote the online classification performances in a target domain. These methods mainly focus on linear models which cannot represent the emotional states, so we adopt hypergraph structure to represent the relationships between emotional states and features by revealing the high-order relationships among data and design online updating to adapt online data.

3 Method

In this work, we perform a hypergraph-based online transfer learning (HOTL) method for ECG-based emotion recognition (ER). The proposed method learns multi-view emotional characteristics with multi-hypergraph and reduces individual differences with hypergraph-based online transfer learning.

The proposed ER method is composed of three parts: feature extraction including statistical features and deep features, hypergraph learning with multi-view features for basic hypergraph model training to enhance the generalization ability of the model for unknown subjects, and hypergraph-based online transfer for online hypergraph model updating to alleviate the influence of individual differences across subjects and nonstationary nature of ECG. The recognition result of an online sample can be obtained from the basic hypergraph model. If the sample is misclassified, the basic hypergraph will be updated by online transfer. The framework of the proposed HOTL is shown in Figure 2.

3.1 Feature Extraction

To classify the emotion states, characteristics of the measured signal must be extracted. In this work, we considered multiview features of ECG signals including time-domain features, frequency-domain features, and deep features.

Time-domain features represent specific changes of ECG signals in time domain. From existing research [Correa et al., 2018], 17 time-domain features were extracted in our paper.

Frequency-domain features are related to the transformation of the signal into the frequency domain and the power of the signal associated with different frequency bands. Similar as [Sep´ulveda et al., 2021], 7 frequency-domain features were extracted in our paper.

To capture the hidden details present in the ECG signal during emotional state change, we employed Resnet [He et al., 2016] to extract more complex features. Specifically, we extract 128-dimensionality deep feature by Resnet.

3.2 Hypergraph Learning withMulti-view Features

We employ a hypergraph model to formulate the known information in multi-view features taking advantage of its highorder correlation modeling. The hypergraph embedding of a hypergraph G is an algorithm to learn feature representations for the vertices that best characterize the similarity relationship [Zhao et al., 2018; Jiang et al., 2019; Gao et al., 2020; Wu and Ng, 2022].

Hypergraph Construct

We generate vertex based on samples and construct multi hyperedges to formulate the information correlation in multiview features. To preserve local information of data and domain-specific information, we construct multi hypergraphs to make samples from the same class close in different views. Specifically, for each class of features, we connect each sample with its k nearest neighbors. The similarity between two vertexes xi and xj are measured by cosine distance

In views of time-domain, frequency-domain, and deep features, the multi hypergraphs are GN = (VN , EN , WN ), where VN is the vertex set, EN is the hyperedge set, and WN is the diagonal matrix of hyperedge weight for the nth hypergraph. So we have GT for time-domain features, GF for frequencydomain features, and GD for deep features. Given the constructed hypergraph GN, we can obtain the incidence matrix HN by computing each entry as:

So multi-view hypergraph is combined with three feature incidence matrices: HT , HF , and HD.

Hypergraph Learning

After constructing the hypergraph of muti-views, hypergraph learning is needed to recognize human emotion. As Zhou's hypergraph algorithm shown [Zhou et al., 2006], the objective function is given by:

where Ω denotes the regular term of the hypergraph structure which is mainly used to control the positional relationship among all vertices on the graph and the consistency of the positional relationship among local vertices.

The Ω is computed as:

where Dv, De denote the vertex degree and the edge degree calculated from diagonal elements of diagonal matrices, H = [HT ,HF ,HD] denotes the multi-view hypergraph incidence matrix.

The Remp(F) is empirical loss term:

where Y is the label matrix, C is the number of classes.

3.3 Hypergraph-based Online Transfer Learning

To classify data in online manner, we denote F = XTM to further recognition online data.

The cost function is composed as:

The hypergraph Laplacian regularizer for M is also under the assumption that strongly connected vertices should have similar labels:

(7)

The empirical loss term on M is defined as:

Φ(M) is a l2,1 norm regularizer to avoid overfiting for M:

To optimize the function, we can reformulate Eq. (6) as:

Note that the l2,1 norm regularizer is convex and nonsmooth. Therefore, as in [Zhang et al., 2018], Eq.(10) is relaxed into the following optimization problem

(11)

where U is a diagonal matrix and the i-th diagonal element is defined as

We initialize U as an identify matrix here empirically. The iteratively reweighted least squares method is employed to solve the problem in Eq.(11). More specifically, we alternatively update each variable while fixing the other until the objective function converges. First, we fix U and derive to M directly. The close-form solution can be written as:

Then we fix M and update U by Eq.(12). The procedure is repeated until U and M are stable.

The basic idea of our HOTL solution is based on the online learning strategy. Consider a new target sample xt with a label yt, we can get the predict label y′t by C(xt) = arg max k xTt M. If y′t = yt, we think that the model get a true result and fix the M. When y′t ̸= yt, the model should be updated to adjust the fault result. In our work, a hypergraphbased online transfer learning method is proposed to update the basic model.

For a target sample xt, if the predict y′t is wrong about the truth, we exploit the label yt to construct a new hyperedge to connect the sample with the basic hypergraph. A straight way is connect the sample with other samples which have the same class. The online transfer learning strategy is to update the basic hypergraph model M by adding a new hyperedge et connecting the online target sample xt:

where Xs denotes the source data of existing subjects, xt denotes an online target sample of a new subject. ∆s = I-D- 1 2 v HD-1 e HT D- 12 v is the normalized hypergraphLaplacian of source data, and δt = 1 - d- 12 v htd-1 e hTt d- 1 2 v is the Laplacian value of the xt, where ht is added into H to con-

nect the new sample.

According to the updating model, for an online target sample xt+1, the category of xt+1 can be obtained by

4 Experiments

4.1 Datasets

AMIGOS is a publicly available database [Correa et al., 2018], which is preferred over other available databases. In this experiments, we construct two binary tasks: negative/positive valence and low/high arousal, where valence represents the emotion is positive or negative, and arousal means the emotional intensity is low or high.

4.2 ComparisonMethods

To show the performance of our method, we compare with some methods:

• Basic classifier: KNN.
• Online learning and online transfer learning method: PA [Crammer et al., 2006], PAIO, HomOTL [Zhao et al., 2014], HomOTLMS [Wu et al., 2017a].
• Emotion recognition method: Naive Bayes [Correa et al., 2018], XGBoost [Tung et al., 2018], BLSTM+Attention[Li et al., 2020], unsupervised domain adaptation (UDA)+online data adaptation (ODA) + SVM [He et al., 2021].

4.3 Experimental Protocol

In this work, we used the recordings of the short and the long experiments together. The ECG signal was then filtered with a band-pass filter between 0.5 and 30 Hz. Then each ECG signal is divided into 20-s segments. To evaluate our proposed method in online setting, we use the leave one subject out cross-validation (LOSO), i.e., one subject is select as the unknown subject and the rest subjects are in training process each experiment.

4.4 Experimental Results and Analysis

Online Emotion Recognition

Experimental results on online emotion recognition for new subject of our HOTL and the compared baselines on Amigos datasets are reported in Table 2-4, respectively.

The results of online emotion recognition compared with basic classifier and online transfer learning methods are shown in Table 2 and Table 3. It can be seen that our proposed HOTL outperforms existing OTL methods. Our HOTL achieves the best recognition accuracy in all subjects. This result reveals that our hypergraph-based online transfer learning method can mine the high-order relationships among data to improve the classification performance.

The results of online emotion recognition compared with existing ER methods are shown in Table 4. It can be seen that our proposed HOTL outperforms existing ER methods. The average accuracy is 0.894 and the F1-score is 0.917 of our HOTL in arousal which gets the best result in ER methods. For valence evaluation, HOTL achieves 0.827 in average accuracy and 0.818 in F1-score.

Figure 3 shows the comparative results about our HOTL and existing methods, regarding emotion classification, we found slightly better performance for arousal than for valence when considering the same type of features and classifier, similar to other works. The good performance of the hypergraph classifier might be due to its characteristics, since the hypergraph structure can represent the high-order relationships among data.

Ablation Study

To understand our method more deeply, we propose to compare in two parts: (1) Hypergraph learning (HL) without online updating. (2) Online emotion recognition results in different combination of features: time-domain (TD) features, frequency-domain (FD) features, Resnet features.

Table 5 shows the average accuracy of the comparisons in online emotion recognition experiment. Classifiers using Resnet features outperform the ones employing traditional time and frequency features. The increased performance can be explained by the fact that deep network can extract features from deep perspective. In addition, HOTL outperforms the hypergraph learning due to the online updating.

Parameter Sensitivity

We evaluate the hyper-parameters of our HOTL, i.e., µ and λ. Experimental results are shown in Fig.4. These results show the parameter sensitivity of HOTL in the emotion recognition. It can be seen that the values of µ and λ are stable in our model. The best results of our model is in µ = 1 and λ = 5.

5 Conclusion

In this paper, we present a hypergraph-based online transfer learning (HOTL) method for online emotion recognition. Our motivations are formulated by two phase, a hypergraph model with multi-view features is trained from existing subjects data and a hypergraph-based online transfer learning method is proposed to adapt incoming data of unknown subject. HOTL can adapt the unknown target data well by the high-order relationships structure and the simple effective updating. Extensive experimental results reveal that our method outperforms the baselines and several state-of-the-art methods.

Acknowledgments

This work is supported in part by a grant from the National Natural Science Foundation of China (No. 61976047), in part by grants from the Science & Technology Department of Sichuan Province of China (No. 22ZDYF2694), and in part by a grant from the Fundamental Research Funds for the Central Universities (No. ZYGX2021YGLH016), and in part by a grant from Sichuan University West China Nursing Discipline Development Special Fund Project (No. HXHL21004).