Daytime Sleepiness Level Prediction Using Respiratory Information

1 Introduction

Inadequate sleep causes daytime sleepiness, and consequently, leads to productivity decline, accidents and health deterioration. Huge economic loss caused by sleepiness was reported in the recent studies [Hafner et al., 2016; Hillman et al., 2018]. To design an effective strategy to reduce daytime sleepiness, the association between sleepiness and lifestyle habits (e.g. sleep duration, activities and food) must be revealed. Subjective measures are widely employed to investigate the cause and effect of sleepiness [ ˚Akerstedt et al., 2014; Shimura et al., 2018]. However, there have been a quite limited number of long-term and large-scale quantitative studies on sleepiness in daily living. If daytime sleepiness can be measured automatically, it will bring a huge impact on the field of health and wellness research from the aspects of time resolution and burdens for both subjects and researchers.

A wide variety of physiological signals including heartbeat, respiration, pupil dilation, electrodermal activity are affected by the autonomic nervous system (ANS) [Robertson et al., 2012]. The relation between sleepiness and ANS has also been confirmed [Pressman and Fry, 1989]. Therefore, a number of technologies have been proposed to detect sleepiness from the physiological information by using this principle. In this paper, we attempt to predict daytime subjective sleepiness by using a wearable respiration sensor. Respiration is one of the promising physiological information to capture various mental states and activities such as sleep and speaking. This approach allows us to track daytime sleepiness automatically for a long period of time.

In order to develop a high performance predictor, we need to tackle two difficulties: The first technical challenge is the appropriate design of an experiment to collect sleepiness data under a natural condition. Most of the previous studies focused on driver's drowsiness and did not consider daytime sleepiness. We conduct a data collection procedure which contains both active and passive tasks representing the natural situations in daily life. The second technical challenge is to train a classifier from a time series data with the class imbalance problem. Sleepiness data collection under the natural situation causes the bias in sleepiness levels. To cope with this problem, we combine a class imbalance handling technique and a time series processing method with a supervised machine learning classifier. To the best of our knowledge, this is the first study that predicts daytime sleepiness by using a wearable respiration sensor.

2 RelatedWork

Various measurement methods are carried out to evaluate sleepiness. Subjective measures such as the visual analog scale [Monk, 1989], the Stanford sleepiness scale [Hoddes et al., 1973], the Epworth sleepiness scale [Johns, 1991] and the Karolinska sleepiness scale (KSS) [ ˚Akerstedt and Gillberg, 1990] are widely employed to measure sleepiness in laboratory and clinical settings. Kaida et al. demonstrated the validity and reliability of KSS against electroencephalogram (EEG) and behavioral indicators of sleepiness [Kaida et al., 2006]. Facial and behavioral expression scores annotated by experts were also used in a lot of automotive drowsiness detection studies [Hachisuka et al., 2011]. As a physiological approach, EEG-based methods are frequently used in laboratory settings [Sforza et al., 2002], although it is difficult to install in daily living situations.

There have been plenty of studies with regard to drowsiness detection for drivers. Physiological and behavioral information such as heart rate variability based on electrocardiogram [Vicente et al., 2016], eye blink [H¨akk¨anen et al., 1999], driver's behavior captured by an accelerometer and a gyroscope in a smart watch [Lee et al., 2016] were employed to detect driver's drowsiness.

Respiration reflects voluntary and involuntary motions, hence it contains the various information about mental states. Sleep stage prediction is one of the promising application [Tataraidze et al., 2015]. Plarre et al. investigated a wearable stress monitoring which utilized respiration and electrocardiogram data obtained in a daily life environment [Plarre et al., 2011]. Furthermore, Igasaki et al. reported that subjective sleepiness while driving can be predicted by respiration [Igasaki et al., 2016]. These previous studies utilized a thoracic respiration band to collect respiratory data.

3 Data Collection

3.1 Experimental Setting

Eighteen individuals (nine males and nine females) who belonged to our affiliation participated in the experiment. The mean of their age was 33.8 and the standard deviation was 9.8. The design of the experiment was approved and conducted according to the ethical guidelines of Toyota Motor Corporation. We sufficiently explained the details and obtained the informed consents from all the subjects.

We did not include the driving task in the experiment as it may undesirably affect subjects' sleepiness. More specifically, they try hard to keep awake because making mistakes while driving may lead to fatal accidents, while they often accept sleepiness and fall sharply into sleep under daily living situations. It has been reported in the literature that this resistance to sleepiness can affect physiological responses [Horne and Reyner, 1999], and consequently, respiratory data while driving and daily tasks may reflect sleepiness in different ways. In order to collect natural sleepiness data in daily living, the experiment was composed of the following two main parts with a five-minute rest in between:

• Typing task: The subjects were instructed to type the content of a book as much as possible for five minutes.
• Video watching task: The subjects were instructed to sit on a sofa and watch a movie for 90 minutes. They had a choice of which one to watch from several dramas and animated movies without stimulating scenes.

We started the experiment at around 14:00 every time to make sure that the subjects gradually got sleepy from the alert state during the experiment. Before the experiment, we instructed them to take enough sleep the day before, not to take caffeine, not to eat too much on the day of the experiment and go to the restroom beforehand. Furthermore, we added the following two procedures to make the subject feel comfortable during and after the experiment: 1) Provide bottled water. The subject can drink it anytime they want. 2) At the end of the experiment, increase the luminance of the room to reduce the subject's sleepiness. All the subjects executed this protocol twice in different days. A quiet room in our laboratory was used as the experimental venue. The overview of the room is shown in Figure 1.

In order to measure their respiratory signal and body movement, we employed the newly developed wearable respiration sensor [Kaji et al., 2018]. The sensor was attached to a subject's abdomen as shown in Figure 2 to capture the small abdominal movement as a respiratory waveform. It is easier and more comfortable to wear than a thoracic band, hence suitable for the data collection in the daily living situation. Three-axis accelerometer built into the sensor was used to measure body movement. We set the sampling frequency to 20 Hz.

As the ground truth of sleepiness, KSS, a 9-point Likert scale rated from "extremely alert (1)" to "very sleepy (9)" [ ˚Akerstedt and Gillberg, 1990], was employed. We programmed it on an Android tablet to periodically ask sleepiness with the notification sound and vibration. The subjects were asked their sleepiness before and after the typing task and during the video watching task every three minutes. We chose the shorter interval of asking KSS than one used in the previous study on driver's drowsiness [ ˚Akerstedt et al., 2005] to increase the number of the available data. However, we confirmed it hardly affected the subject's sleepiness in the preliminary experiment. In fact, we observed the subject sometimes did not notice the sound and vibration, and therefore could not answer KSS due to strong sleepiness. Missing values were replaced with KSS 9 in such a case.

The collected KSS scores were then divided into two sleepiness levels, namely, Alert (KSS 1 to 5) and Sleepy (KSS 6 to 9) as shown in Table 1. We reframed the sleepiness prediction as a binary classification problem because the physiological data were heavily noisy and the subjective measure could have a bias depending on a subject. Binary classification is much simpler in terms of resolution than multiclass classification and regression, but still useful for capturing the change points and the trends of sleepiness.

Those data were synchronized and stored in the data server. As a consequent, we collected approximately 3,420 minutes of data in total.

3.2 Data Processing

Preprocessing

The collected respiratory data were first preprocessed with a fourth-order butterworth bandpass filter for noise reduction. We set the lower bound of the passband frequency to 0.05 Hz, which was sufficient to capture respiratory signals given that the breathing rate of healthy adults ranges from 12 to 20 per minute, i.e. 0.2 - 0.33 Hz [Yuan et al., 2013]. The upper bound frequency was chosen so that the filtered waveforms reflect important features of raw waveforms. We found from the preliminary experiment that 2.5 Hz was the optimal frequency for balancing noise reduction and the preservation of the original waveform features.

Both of the filtered respiratory data and the raw acceleration data were then segmented into 60-second windows with 30-second overlaps. A sleepiness level is associated with fluctuations in a breathing pattern, therefore each window has to be long enough to contain as many breathing cycles as required for the accurate evaluation of breathing fluctuations. The 60-second window can contain at least six breathing cycles, which were found to be necessary from our preliminary experiment. We annotated each 60-second window with the nearest value of the collected subjective sleepiness levels.

Feature Extraction

We calculated the mean, standard deviation, max and min of nine respiration parameters described as follows for each 60second window. The number shows the unit's place of the feature ID.

1. Inspiratory Duration: The duration between the bottom and the next peak of a respiratory waveform.
2. Expiratory Duration: The duration between the peak and the next bottom of a respiratory waveform.
3. Duration Ratio: The ratio of Expiratory duration to Inspiratory duration.
4. Pause Duration: The duration of the rest between expiration and inspiration.
5. Stretch: The amplitude from the peak to the bottom.

1. AUIC: The area under an inspiratory curve.
2. AUEC: The area under an expiratory curve.
3. Area Ratio: The ratio of AUEC to AUIC.
4. Peak-to-Peak Interval: The interval between the peak and the next peak of a respiratory waveform.

Figure 3 illustrates the respiration parameters. Pause Duration is geometrically defined as shown in Figure 3. To be more specific, find the point which is as high as half of the Stretch on an expiratory curve, and let HD denote the duration between the peak and this point. Pause Duration (PD) can be defined as the following equation using Expiratory Duration (ED): PD = ED-2HD. We did not use frequency-domain features in order to suppress the computational cost as we aim to develop the online sleepiness prediction and embed it into the wearable sensor.

We also computed the mean, standard deviation, max and min of the ℓ2 norm of the first-order difference of three-axis accelerations ||at - at-1|| as the body-movement features, where at = (axt , ayt , azt )⊤ is the t-th output. We numbered this body-movement parameter 9. All the extracted features were normalized to have zero mean and unit standard deviation to reduce the inter- and intra- personal differences of breathing patterns and body motion.

Feature Selection

We took a two-step method for the feature selection. The first step was to eliminate strong correlation among the features. We calculated the correlation coefficients of all the pairs, and identify the pairs/groups of the features which strongly correlated with each other (if the absolute value of the correlation coefficients exceeded 0.7). All the features in the pairs/groups except one with the highest correlation with the sleepiness level were removed. It resulted in removal of 19 features.

In the second step, we employed the aggregated feature importance computed by multiple random forests (RFs) [Breiman, 2001] as a selection criterion because it was reported to be able to well balance the robustness of selected features and prediction performance [Saeys et al., 2008]. We created 100 RFs and simply aggregated the feature importance derived from each RF. Undersampling was used before creating each RF to take into account the imbalanced dataset (see the next section for the detailed explanation of the class imbalance problem). The candidate features were ranked based on the importance and we included the best four features in our experimental models. The aggregated importance of the 21 candidate features is shown in Figure 4.

As a result of the above selection method, the mean of Inspiratory Duration, the mean of Expiratory Duration, the min of Duration Ratio and the min of the body-movement parameter were selected. The selected set included three features related to Inspiratory Duration and Expiratory Duration. They were also selected in the feature selection algorithm performed in the previous study which built models for stress prediction in a natural environment [Plarre et al., 2011]. These results indicate Inspiratory Duration and Expiratory Duration are of great importance in predicting human mental states in an everyday life situation. On the other hand, none of the features related to the amplitude (AUIC, AUEC and Stretch) were effective mainly because of the measuring principle of the sensor. It measures respiration by sensing the abdominal movements, which leads to large variability in the amplitude depending on the subject's posture and body size.

4 Method

Let x = (x1, · · · , xd)⊤ be a d-dimensional respiration and body-movement feature vector and y ∈ {Alert, Sleepy} be the class label of the sleepiness levels determined by KSS. There are various problems when predicting human mental states from physiological data by using supervised classification methods. We explored a procedure to improve the performance of the sleepiness prediction by focusing on the two major problems, i.e. class imbalance and the i.i.d. assumption. The effectiveness of the proposed procedure was examined through careful model validation.

4.1 Class Imbalance Handling

Class imbalance often becomes a serious problem in the domain of human mental state prediction because mental states are not completely controllable even if experiments are designed to induce desired states to some extent. Sleepiness is not an exception to this and the collected data were imbalanced (Alert: 30.6%, Sleepy: 69.4%). Learning a classifier from imbalanced datasets without proper techniques can result in biased prediction towards the majority class.

A number of methods for handling the class imbalance problem have been developed and proposed. Some experimental studies demonstrated that undersampling bagging worked better than other types of methods [Galar et al., 2012; Khoshgoftaar et al., 2011]. In this paper, we used two types of undersampling bagging i.e. exactly balanced bagging (EBB) [Chang et al., 2003] and roughly balanced bagging (RBB) [Hido and Kashima, 2008] in addition to the other basic methods i.e. undersampling (US), oversampling (OS) and synthetic minority oversampling technique (SMOTE) [Chawla et al., 2002] to compare the effectiveness.

4.2 Time Series Processing

Although supervised classification methods such as support vector machine (SVM) [Vapnik, 1998] and RF can well represent the relation between feature vectors and classes, the sequence of prediction results as a whole can be unnatural due to the i.i.d. assumption. Especially in the case of using a wearable sensor, data can be easily affected by body motion, making it difficult to obtain noiseless waveforms even after filtering. Consequently, supervised classifiers predict the uninterpretable sequence of sleepiness which is unlikely to occur in real life.

We combined classification methods and hidden Markov model (HMM) [Rabiner, 1989] to overcome this problem like [Han and Wang, 2014; Richard et al., 2018] who combined a neural network and HMM. In their procedure, the posterior probability p(y|x) of each class is computed by a classifier, and converted to the likelihood which is proportional to a conditional probability:

where p(y) is a class prior. The likelihood is used in the Viterbi decoding [Rabiner, 1989] to predict the sequence of the classes. In this experiment, p(y) was empirically computed from the ratio of classes contained in a training set.

4.3 Procedure for Sleepiness Prediction

Considering the aforementioned problems, we established the procedure for the sleepiness prediction illustrated in Figure 5. Both a class imbalance method and HMM are combined with a supervised classifier at the same time. For training, we first applied a class imbalance method to obtain a balanced dataset, and then built an unbiased classifier. The parameters of HMM such as the state transition probabilities are estimated from the sequence of the subjective sleepiness levels. Prediction is done in two steps using the trained classifier and the HMM parameters.

4.4 Experiment

Using the selected four features, we conducted an experimental analysis of the sleepiness prediction with respect to the classifiers, the class imbalance handling and the time series processing. All the combinations were evaluated based on the macro F-measure expressed as the equation below since the dataset was imbalanced;

where c = 2 is the number of the classes. The related works employed the k-fold cross validation to evaluate the performance of their models, which means the training set and the test set included the data obtained from the same subject [Igasaki et al., 2016; Plarre et al., 2011]. However, physiological data with corresponding labels are often quite hard to collect from all system users in practice. We used the leave-one-subject-out cross validation (LOSO-CV) [Bao and Intille, 2004] to ensure the evaluation under more practical conditions. In our LOSO-CV, the samples of 17 subjects were used for training the model and the remaining one subject's samples were used as the test set.

The baseline classifiers we selected were naive Bayes (NB), SVM, logistic regression (LR), RF and a three-layer neural network (NN). The hidden layer of NN consisted of four nodes with the sigmoid function. Note that all of these classifiers can produce the posterior probabilities of the sleepiness levels. Here, the method proposed in [Platt, 2000] was employed in order to yield the posterior probabilities from SVM. We also compared a median filter (MED) of a 150-second window with HMM as a technique for smoothing time series data. Since random sampling was used in the class imbalance handling, we executed 100 trials with different random seeds to evaluate the performances.

5 Result

Table 2 presents the mean and standard deviation of an Fmeasure for each combination over 100 trials. Bold face denotes the best and comparable combinations in terms of the class imbalance handling techniques according to the paired t-test at the significance level of 5%. We first compared the results with and without the feature normalization (the first two columns). Although there were some combinations which showed the better F-measure without the normalization, it generally worked well. All the baseline classifiers with HMM got the significant improvement by the feature normalization. This result indicates that the normalization was especially helpful for the accurate prediction of the posterior probabilities. In light of this, we used the normalized features to compute the F-measure of the combinations with the class imbalance handling.

The results shown in Table 2 proved that the proposed procedure was effective for all the five classifiers. It was able to improve the F-measure by 0.1109 on average compared to the single classifier. The best performing combination was LR-EBB-HMM with the mean F-measure of 0.7059, which was rather high considering the ground truth of sleepiness was based on the self-report ratings.

The improvement made by adding a class imbalance handling technique and HMM also can be confirmed visually.

The transitions of the predicted sleepiness by three different combinations are shown in Figure 6. LR-EBB-HMM in Figure 6c correctly predicted the trend of the sleepiness transition while LR in Figure 6a was strongly biased towards the Sleepy class. However, LR-EBB-HMM seems to have a tendency to ignore relatively small changes in sleepiness. LR-EBB in Figure 6b is still not enough because its prediction unrealistically changed in a short period of time, making it difficult to figure out how the actual sleepiness changed over time.

We observed that adding HMM alone significantly improved the F-measure. This may be because posteriors are divided by priors when put into HMM, which is substantially equal to adjusting the class distribution of a dataset. In this respect, the combination of supervised classifiers and HMM also acts like a class imbalance method. Indeed, LR-HMM, RF-HMM and NN-HMM outperformed some other combinations with the class imbalance method.

When focusing on the class imbalance handling, we found EBB and RBB were included in many of the best or comparable combinations, which was consistent with the previous studies arguing that undersampling bagging was an effective approach. We confirmed these bagging-based methods worked well also with HMM and a median filter.

Figure 7 shows the mean F-measure of LR-EBB-HMM

by subject. The F-measure varies across the subjects and it seems hard to build a common high-performance predictor for all. This result shows that there were huge interpersonal differences in breathing patterns like many other physiological indicators. One effective approach to this problem might be to gradually adapt a model to each user as accumulating his/her data.

6 Conclusion

In this study, we collected a set of respiration and acceleration data associated with subjective sleepiness levels using a novel wearable sensor. Considering the characteristics of the obtained dataset, we proposed combining a class imbalance technique and HMM with a supervised machine learning classifier. The performance of the models was examined through the experiment in a careful manner.

The major contribution of this study lies in our methodological approach for predicting the subjective sleepiness levels in daily living. We collected the physiological data of the subjects doing both active and passive tasks which represent activities in everyday life, unlike the prior studies which concentrated on driver's sleepiness. Therefore, our dataset and derived models more precisely reflected the actual association of the physiological information and sleepiness in daily living. We found from our experiment that combining a class imbalance technique and HMM significantly enhance the prediction performance of a machine learning classifier. The Fmeasure of our best model was 0.7059, which was sufficient to capture the trend of sleepiness transition. Our procedure also has the potential to be utilized in a broad range of applications in the field of human mental state prediction because class imbalance and time series are the common properties of data in the field.

This study also has some limitations. First of all, only 18 subjects participated in the data collection. More samples are necessary for more accurate sleepiness prediction as well as better understanding of the association between physiological features and sleepiness. Secondly, there remain a lot of activities to consider other than typing and watching a movie. We need to investigate how breathing patterns and sleepiness shift during different tasks in order to create a robust model as respiration can be affected by actions and feelings.

The future work may include a study on the relation between sleepiness and various performance indicators such as productivity, health, etc. since the wearable-sensor-based approach enables the long-term tracking of sleepiness without burdening users. We also need to establish a methodology for individual adaptation of a sleepiness prediction model because we found the interpersonal difference in the respiratory information and the self-report ratings made it difficult to build one high-performance predictor for all users. However, collecting the sufficient amount of labeled data is generally hard due to the high annotation cost. Some methods from weakly supervised learning [Zhou, 2017] and online learning will be helpful for this purpose.