SalientSleepNet: Multimodal Salient Wave Detection Network for Sleep Staging

1 Introduction

Sleep staging is important for the assessment of sleep quality and the diagnosis of sleep disorders. To determine sleep stages, sleep experts use electrical activity recorded from sensors attached to different parts of the body. The recorded signals from these sensors are called polysomnography (PSG), consisting of electroencephalogram (EEG), electrooculogram (EOG), and other physiological signals. These recorded signals are divided into 30s sleep epochs and sleep experts classify them into five different sleep stages (W, N1, N2, N3, and REM) according to the American Academy of Sleep Medicine (AASM) sleep standard [Iber et al., 2007]. However, this manual approach is labor-intensive and timeconsuming due to the need for PSG recordings from several sensors attached to subjects over several nights. Therefore, many researchers attempt to develop automatic sleep staging methods.

∗ corresponding author: wj@bjtu.edu.cn

To automate sleep staging, some deep learning methods are applied to achieve state-of-the-art performance [Phan et al., 2019; Perslev et al., 2019; Supratak and Guo, 2020]. However, there are several challenges remain open:

1) The salient waves are directly captured from the raw signals. According to the AASM sleep standard, different sleep stages usually have different salient waves in physiological signals. For example, Figure 1 shows that the salient waves of N2 stage are spindle waves and K-complex waves, while the salient wave of N3 stage is delta waves. Most of the existing works use feature extraction to indirectly capture salient waves [Phan et al., 2019; Phan et al., 2020; Chriskos et al., 2019; Hassan and Bhuiyan, 2016]. For example, the time-frequency features of the signals can reflect the saliency wave features to a certain extent. The raw physiological signals are converted into time-frequency images as input to deep learning models for sleep staging [Phan et al., 2020; Hassan and Bhuiyan, 2016]. Although the above methods indirectly capture signal wave features, manually extracted features not only require prior knowledge but may also cause partial information loss.

2) Multi-scale sleep transition rules are utilized effectively. During sleep, the human brain undergoes a series of changes among different sleep stages. These changing patterns are summarized as transition rules in the sleep standard. Experts often determine the current sleep stage based on these rules, combined with its neighboring sleep stages. Figure 2 shows the sleep transition rules have multi-scale characteristics in the AASM standard. To capture sleep transition rules, some mixed models including RNNs are usually used [Supratak et al., 2017; Supratak and Guo, 2020; Mousavi et al., 2019; Phan et al., 2020]. However, most of the existing works ignore to explicitly capture the multi-scale characteristics of sleep transition rules. In addition, these recurrent models are difficult to tune and optimize. It has been found that for many tasks RNNs models can be replaced by feed-forward systems (such as CNN variants) without sacrificing accuracy [Bai et al., 2018].

3) Different modalities have different contributions to distinguish the sleep stages. Figure 1 shows that the EEG waves in the REM and N1 stages are similar. However, the EOG waves in the two stages are quite different. Hence, the contribution of EOG signals to classify REM and N1 stages is greater than that of the EEG signals. On the contrary, the classification of N2 and N3 stages is mainly according to the salient waves in EEG signals. Therefore, when identifying different sleep stages, the required modalities are different. To utilize these multimodal signals, researchers usually pay attention to the multimodal complementarity and merge the multimodal features through concatenate operations [Chambon et al., 2018; Sokolovsky et al., 2019; Phan et al., 2019; Phan et al., 2020]. This ignores that the contribution of each modality to the identification of specific sleep stages is different.

To address the above challenges, we propose a novel deep learning model called SalientSleepNet, a sleep staging model based on multimodal sleep data (EEG signals and EOG signals). Different frommany previous works, the SalientSleepNet can simultaneously capture the salient features of multimodal data and multi-scale sleep transition rules from the raw signals.

Overall, the main contributions of the SalientSleepNet for sleep staging are summarized as follows:

• We develop the U2 -structure stream, which is composed of multiple nested U-units, to detect the salient waves in physiological signals. It is inspired by the popular U2 - Net originally proposed in computer vision.
• We design a multi-scale extraction module to capture sleep transition rules. It is composed of multiple dilated convolutions with different receptive fields for capturing multi-scale rules.
• We propose a multimodal attention module to adaptively capture valuable information from different multimodal data for the specific sleep stage.
• Experimental results show that the SalientSleepNet achieves state-of-the-art performance. In addition, our model parameters are the least compared with existing deep neural networks for sleep staging.

2 RelatedWork

In recent years, time series analysis has attracted the attention of many researchers [Jia et al., 2020b; Jia et al., 2019; Jia et al., 2020a]. Sleep signal as a typical physiological time series helps to diagnose sleep disorders. In earlier research, machine learning methods are utilized to classify sleep stages. However, these methods rely on hand-engineered features that require a lot of prior knowledge. Therefore, the researchers have turned to use deep learning methods for automatic sleep staging [Jia et al., 2020c; Jia et al., 2021; Cai et al., 2020].

At present, deep learning methods based on CNN or RNN have been widely used in sleep staging. For example, DeepSleepNet [Supratak et al., 2017] utilizes CNN to extract time-invariant features, and Bi-directional Long Short-Term Memory (BiLSTM) to learn the transition rules among sleep stages. SleepEEGNet [Mousavi et al., 2019] can extract time-invariant features from the raw signals and capture long short-term context dependencies. XSleepNet [Phan et al., 2020] presents a sequence-to-sequence sleep staging model that is capable of learning a joint representation from both raw signals and time-frequency images effectively. In addition, to lighten the existing models such as DeepSleepNet, TinySleepNet is proposed [Supratak and Guo, 2020] for sleep staging. SleepUtime [Perslev et al., 2019] is a fully feedforward deep learning approach to physiological time-series segmentation. It maps sequential inputs of arbitrary length to class label sequences on a freely chosen temporal scale.

The existing models achieve high performance for sleep staging. Unfortunately, these deep learning methods cannot effectively detect and fuse the salient waves in raw multimodal signals. In addition, the multi-scale transition rules among sleep stages cannot be captured explicitly.

3 Preliminaries

The proposed model processes a sequence of sleep epochs and outputs a predicted label sequence. Each sleep epoch is defined as x ∈ Rn×C , where n is the number of sampling points in an epoch, and C denotes the channels of sleep epoch (i.e. an EEG channel and an EOG channel in this work).

The input sequence of sleep epochs is defined as S = {x1, x2, . . . , xL}, where xi denotes a sleep epoch (i ∈ [1, 2, · · · , L]) and L is the number of sleep epochs.

The sleep staging problem is defined as: learning a mapping function F based on multimodal salient wave detection network which maps sleep epoch sequence S into the corresponding sequence of sleep stages ˆ Y , where ˆ Y = {ˆ y1, ˆ y2, . . . , ˆ yL} and ˆ yi is the classification result of xi. Following the AASMmanual, each ˆ yi ∈ {0, 1, 2, 3, 4} matching with the five sleep stages W, N1, N2, N3, and REM, respectively.

4 Multimodal Salient Wave Detection Network for Sleep Staging

The overall architecture of SalientSleepNet is shown in Figure 3. We summarize five key ideas of SalientSleepNet: 1) Develop a two-stream U2 -structure to capture the salient waves in EEG and EOG modalities. 2) Design a multi-scale extraction module by dilated convolution with different scales of receptive fields to learn the multi-scale sleep transition rules explicitly. 3) Propose a multimodal attention module to fuse the outputs from EEG and EOG streams and strengthen the features of different modalities which make greater contribution to identify certain sleep stage. 4) Improve the traditional pixel-wise (point-wise) classifier in computer vision into a segment-wise classifier for sleep signals. 5) Employ a bottleneck layer to reduce the computational cost to make the overall model lightweight.

4.1 Two-StreamU2 -Structure

Human experts classify the sleep stage mainly based on the salient waves in EEG and EOG signals such as spindle waves, K-complexes, and sawtooth waves [Iber et al., 2007]. Existing sleep staging models extract salient wave features indirectly by converting raw signals into time-frequency images. This may cause information loss and require prior knowledge.

To capture the salient waves in raw EEG and EOG signals directly, we design a two-stream U2 -structure to capture the features of different signals as shown in Figure 3. Specifically, The EEG signals and EOG signals are input into two independent U2 -structures to learn the distinctive features. Each U2 -structure is an encoder-decoder structure and is composed of multiple nested U-units. This is inspired by U2 -Net in salient object detection of images [Qin et al., 2020]. Specifically, each U-unit has three components: a channel-reshape layer, a U-like structure, and a residual connection:

1. Given a 1D feature map X, the channel-reshape layer transforms it into an intermediate feature map to control the channel number of the U-unit:

where Reshape denotes the channel-reshape operation and Xm denotes the intermediate feature map.

1. The U-like structure encodes and decodes the intermediate feature map and obtains X′m:

where Ul denotes the U-like structure with depth of l (l = 4 in our model).

1. The residual connection [He et al., 2016] fuses feature map Xm and X′m with addition operation to reduce the

degradation problem in the deep network:

where X′′m is the output of the U-unit.

Multiple U-units make up the U2 -structure. Specifically, there are 5 U-units in the encoder and 4 U-units in the decoder for each U2 -structure.

4.2 Multi-scale ExtractionModule

Sleep transition rules is important for sleep staging. The transition rules have the characteristics of multiple scales: short scale, middle scale, and long scale. Previous works use the RNN to learn the sleep transition rules. However, the RNN models are difficult to tune and optimize [Bai et al., 2018].

To solve the above problems, we design a Multi-Scale Extraction module (MSE) to capture multi-scale sleep transition rules explicitly, as shown in Figure 4. The MSE has dilated convolutions with different dilation rates to capture features in different scales of receptive fields [Yu and Koltun, 2015]. Specifically, we employ 4 dilated convolutions with dilation rates from 1 to 4 to transform the same input feature map. Then, the feature maps learned from different scales of receptive fields are concatenated to obtain a multi-scale feature map, which is defined as:

where X′′m is the input feature map, DConvr is the dilated convolution with dilation rate r, Xr d is the output of dilated convolution DConvr, and Xms is the multi-scale feature map.

Besides, to reduce the parameters of our model, we apply a bottleneck layer [He et al., 2016] between the encoder and decoder (i.e. implemented in MSE). It reduces the channels of the concatenated feature map which makes the model lightweight and composed of two convolution operations. The bottleneck layer is defined as:

where Bottleneck is the operation of bottleneck layers and Xb is the final multi-scale feature map with channel reduction operation obtained from the MSE. Xb has Cout channels and Cout = Cin/rate, where Cin is the channel number of Xms and rate is the down-sampling rate of bottleneck layer.

4.3 Multimodal AttentionModule

Different modalities have distinctive features which contribute to classifying specific sleep stage. However, existing models ignore that different modalities have different contributions to identify certain sleep stage.

To strengthen the features of different modalities which have greater contribution to classify specific sleep stage in an adaptive approach, we propose a MultiModal Attention module (MMA). As shown in Figure 5, The MMA has two main components: The modality fusion component for fusing the feature maps from two streams and the channel-wise attention component [Hu et al., 2018] for strengthening the most important features in an implicit way for certain sleep stage.

1. Inspired by the fusion method of [Fu et al., 2020], the modality fusion component is defined as:

where Xfuse is the fused feature map and ⊙ is the element-wise multiplication. XEEG and XEOG are the feature maps learned from EEG and EOG streams, respectively.

1. The Xfuse is input into the channel-wise attention component to strengthen the important channels of the fused feature map:

where GAP denotes the global average pooling operation, FCi denotes i-th fully connected layers, δ is the ReLU activation function, σ is the sigmoid activation function, Xatt is the intermediate feature map, and X′att is the output of this component.

4.4 Segment-wise Classification

Existing salient object detection models in computer vision are pixel-wise (point-wise) classification [Qin et al., 2020]. These models can not be applied directly to physiological signal segment-wise classification.

Hence, we design a segment-wise classifier which maps the pixel-wise feature map to a segment-wise predict label sequence. Specifically, an average pooling layer is applied to reduce the size of 1D feature map from X′att ∈ RL′ to Xpool ∈ RL, where L′ = L ∗ n, L is the number of sleep epochs, n represents the number of sampling points in an epoch. Then, a convolution layer with Softmax activation is used to reduce the channel dimension of Xpool and map vectors to predict label sequence ˆ Y .

5 Experiments

We evaluate the performance of SalientSleepNet on SleepEDF-39 and Sleep-EDF-153 datasets [Kemp et al., 2000; Goldberger et al., 2000].

5.1 Datasets and Preprocessing

Sleep-EDF-39 consists of 42308 sleep epochs from 20 healthy subjects (10 males and 10 females) aged 25-34. Each subject contains two day-night PSG recordings except subject 13 whose one recording is lost due to device failure. SleepEDF-153 consists of 195479 sleep epochs from 78 healthy subjects aged 25-101. Similar to Sleep-EDF-39, each subject contains two day-night PSG recordings except subjects 13, 36, and 52 whose one recording is lost due to device failure.

For a fair comparison with other methods, we use the same preprocessing and data for all models. Specifically, each 30s epoch of the recordings is labeled into one of six categories {W, N1, N2, N3, N4, REM}. We merge N3 and N4 stages into a single N3 stage according to the latest AASM sleep standard [Iber et al., 2007] and remove before and after in-bed epochs for only in-bed recordings are used as recommended in [Imtiaz and Rodriguez-Villegas, 2014]. We adopt Fpz-Cz EEG and ROC-LOC EOG channels in this study. All signals have a sampling rate of 100 Hz.

5.2 Baseline Methods

We compare the SalientSleepNet with the following baselines:

• SVM [Hearst et al., 1998]: Support Vector Machines uses a Gaussian kernel function for sleep staging.
• RF [Breiman, 2001]: Random Forests are an ensemble learning method for sleep staging.
• DeepSleepNet [Supratak et al., 2017]: DeepSleepNet utilizes CNN to extract time-invariant features and BiLSTM to learn the transition rules among sleep stages.
• SeqSleepNet [Phan et al., 2019]: SeqSleepNet is composed of parallel filterbank layers for preprocessing the time-frequency images and bidirectional RNN to encode sleep sequential information.
• SleepEEGNet [Mousavi et al., 2019]: SleepEEGNet extract time-invariant features from the sleep signals and capture long short-term context dependencies
• SleepUtime [Perslev et al., 2019]: SleepUtime maps sequential inputs of arbitrary length to sequences of class labels on a freely chosen temporal scale.
• TinySleepNet [Supratak and Guo, 2020]: TinySleepNet is a mixed model of CNN and unidirectional RNN.

5.3 Experiment Settings

We implement the SalientSleepNet based on the TensorFlow framework. Our model is trained by Adam optimizer with learning rate η = 10-3 . The training epoch is 60 and batch size is 8. The input sleep epoch sequence length L is 20. Besides, the down-sampling rate of the bottleneck layer rate is 4. We apply 20-fold cross-validation to evaluate the crosssubject performance of SalientSleepNet. Other hyperparameters are given in our code, which is publicly available at https://github.com/ziyujia/SalientSleepNet.

5.4 Experiment Results

We compare SalientSleepNet with other seven baseline methods on two publicly available datasets as shown in Table 1. The results show that SalientSleepNet achieves the best overall performance compared with other baseline methods.

The traditional machine learning models (SVM and RF) can't capture various features. Some mixed deep learning models such as DeepSleepNet, SeqSleepNet, and TinySleepNet combine CNN to learn the features in sleep epochs and RNN to capture transition rules between sleep epochs. Therefore, These models achieve better performance than traditional machine learning models.

Although the mixed models can achieve high accuracy, these models are difficult to tune and optimize. In addition, existing models do not make full use of salient wave features in different modalities and some models require timefrequency images as inputs which may cause partial information loss. Different from previous works, our model can simultaneously capture the salient features of multimodal data and multi-scale sleep transition rules from the raw signals. In addition, our model strengthens the contribution of different modalities features to classify different sleep stages. Therefore, SalientSleepNet achieves better overall performance than other baseline methods. It is worth noting that the N1 stage usually has the smallest sample number and is easy to confuse with REM stage. However, our model improves the F1-score for N1 stage to a certain extent. Besides, our model has the minimum parameters compared with other deep learning models.

We visualize the point-wise outputs of U2 -structure as shown in Figure 6. It illustrates that our model can detect salient waves in multimodal signals to a certain extent. For example, in EEG signals, the sleep spindle wave and K-complex are in N2 stage. The delta wave appears in N3

stage. Besides, in EOG signals, the slow eye movement wave and rapid eye movement wave exist in N1 stage and REM stage, respectively. Overall, our model can detect these salient waves, which makes this model interpretable to a certain extent.

5.5 Ablation Experiment

To evaluate the effectiveness of each module in our model, we design several variant models:

• U structure (basic): This model is a two-stream U structure without nested U-units, MSE, and MMA. The feature maps obtained from two streams are concatenated for sleep staging.
• U2 structure: The nested U-units are applied to the basic model.
• U2 structure+MSE: This model adds the multi-scale extraction modules based on U2 structure.
• U2 structure+MSE+MMA (SalientSleepNet): This model adds the multimodal attention module instead of a simple concatenate operation based on U2+MSE model.

Figure 7 presents the U2 -structure obviously improves the performance of basic model, because it captures more diverse salient waves. Then, the structure with MSE learns the multiscales features which improve the performance. In addition, the MMA is added to adaptive capture the important features of different modalities for certain sleep stage.

Besides, to evaluate the effectiveness of the bottleneck layer in reducing model parameters, we removed this layer in MSE. Compared with the model without bottleneck layer which has 1,128,865 parameters, SalintSleepNet only has 948,795 parameters. The bottleneck layer reduces the number of parameters in the model by 16%. It allows our model to keep the smallest size while achieving the best performance than other baseline methods at the same time.

6 Conclusion

In this paper, we propose a multimodal salient wave detection network for sleep staging. Our work is the first attempt to borrow the U2 -Net from the visual saliency detection domain to sleep staging. SalientSleepNet can not only effectively detect and fuse the salient waves in multimodal data, but also extract multi-scale transition rules among sleep stages. Experiment results show that SalientSleepNet achieves state-ofthe-art performance. The parameters of our model are the least among the existing deep learning models. In addition, the proposed model is a general-framework for multimodal physiological time series.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities (Grant No.2020YJS025) and the China Scholarship Council (Grant No.202007090056).