A Novel Distribution-Embedded Neural Network for Sensor-Based Activity Recognition

1 Introduction

Human activity recognition is an important technique with a wide range of real-world applications, e.g., assisted living, personalized health monitoring, security and smart homes [Janidarmian et al., 2017; Bulling et al., 2014; Lara and Labrador, 2013]. The recognition of human activities has generally been approached in two categories based on different types of involved sensors, namely ambient-sensor-based and wearable-sensor-based. The former utilizes devices fixed in locations of interest, such as video cameras or WiFi access points attached to a fixed desk or wall. The latter refers to sensors attached to a human body [Chen et al., 2012]. In this work, we focus on wearable-sensor-based activity recognition scenarios since wearable on-body sensors alleviate environment constraints and are free from privacy issues (take cameras as example, all the behaviours of participants are recorded and are easily recognized by others, while signals of wearables are not visible) [Yang et al., 2015]. In these scenarios, the common practice is that the continuous multivariate time series data is partitioned into segments first, and each segment is assigned a specific activity. Feature extraction is commonly conducted on each segment to extract discriminative features. The extracted features are then fed into a classifier or fully-connected layers to recognize different activities [Hammerla et al., 2016].

Existing feature extraction approaches can be classified into two categories: feature-engineer-based and deeplearning-based. The approaches of the former category aim to extract various aspects of information underlying each sensor-reading segment, such as statistical information [Janidarmian et al., 2017], meta information, e.g., overall shape and spatial information [Lara and Labrador, 2013; Hammerla et al., 2013; Lin et al., 2007]. These approaches usually require domain knowledge to manually design proper features for specific applications, which is labor-intensive and time consuming. To overcome the limitations of featureengineering-based approaches, Qian et al. [2018] proposed the SMMAR method to automatically extract all orders of moments as statistical features by using kernel embedding technique of distributions. However, SMMAR fails to extract temporal and spatial information from the segments of sensor readings, which is important for recognizing activities.

The approaches of the latter category aim to design deep neural networks to extract temporal and/or spatial features from the segments of sensor readings automatically [Wang et al., 2017]. Different types of neural networks have been proposed to extract different kinds of information [Hammerla et al., 2016]. Basically, deep feed-forward networks (DNNs) are used to extract higher-level features without taking temporal or spatial information into consideration. Convolutional neural networks (CNNs) are used to extract locally translation invariant features with respect to the precise location or precise time of occurrence of certain pattern within a data segment [Zeng et al., 2014; Yang et al., 2015; Ignatov, 2018]. Recurrent neural networks (RNNs) are suitable for exploiting the temporal dependencies within the activity sequence. The state-of-the-art for sensor-based activity recognition are basically combinations of these three types of base models [Morales and Roggen, 2016]. Though deeplearning-based approaches are able to learn powerful features to represent temporal and/or spatial information underlying sensor data, they fail to capture statistical information, such as different orders of statistical moments, which has proven to be useful for activity recognition [Qian et al., 2018; Qian et al., 2019].

In this paper, we propose a novel Distribution-Embedded Deep Neural Network (DDNN) for wearable-sensor-based human activity recognition. The main novelty of our network lies in that we encode the idea of kernel embedding of distributions into a deep architecture, such that besides temporal and spatial information, all orders of statistical moments can be extracted as features to represent each segment of sensor readings, and further used for activity classification in an end-to-end training manner. Compared with the SMMAR method [Qian et al., 2018], which also makes use of the kernel embedding technique to extract statistical features for sensor data, our proposed DDNN is capable of learning more powerful features beyond statistical features. In addition, SMMAR assumes that all activities are segmented beforehand, while DDNN relaxes the perfect-segmentation assumption by simply using sliding windows, which makes DDNN more practical for real-world scenarios. Moreover, SMMAR uses a single kernel to embed distributions, which may be sensitive to the parameter settings of the kernel, while DDNN uses a deep neural network to approximate the feature map of the kernel, which is more flexible as the parameters of the deep neural network are learned from the data.

In summary, our contributions are two-fold:

• Our proposed DDNN is a unified end-to-end trainable deep learning model, which is able to learn different types of powerful features for activity recognition in an automated fashion.
• Extensive experiments are conducted on several benchmark datasets to demonstrate the superior performance of our proposed DDNN.

2 RelatedWork

Feature-engineering-based machine learning methods. These methods include PCA, LDA, basis transform coding (wavelet transform and Fourier transform) and handcrafted statistical features of raw signals including orders of moments (mean, variance, skewness, etc), median, etc [Janidarmian et al., 2017]. SMMAR method [Qian et al., 2018] automatically extracts all orders of moments as statistical features by using kernel mean embedding technique. Besides statistical features, several methods treat extra meta information as extra structural features. For instance, ECDF method preserves the overall shape and spatial information of time series data [Hammerla et al., 2013]; SAX method transforms continuous data into symbolic representations [Lin et al., 2007].

Deep learning methods. Deep learning methods can substitute for the manual feature design and extraction procedure. The first deep learning method on activity recognition applies Restricted Boltzmann Machines (RBMs) to compare with manual features [Pl¨otz et al., 2011]. CNNs are the most widely used frameworks in this field [Zeng et al., 2014; Yang et al., 2015; Ignatov, 2018]. Yang et al.[2015] customized CNNs along temporal dimension of activity data to extract salient patterns of sensor signals at different time scales. Besides, temporal dependencies in time-series data are proven to be beneficial for activity recognition as well. DeepConvLSTMmodel [Morales and Roggen, 2016] applies two Long Short-TermMemory (LSTMs) layers on top of the abstract feature representations extracted by four convolutional layers. There are also research works to jointly learn shallow features by traditional methods and deep features by deep models [Rav`ı et al., 2017; Ignatov, 2018]. There are also attempts of combinations of shallow classifiers with deep learned features. Hammerla et al. [2016] provided systematic comparisons on the performance of state-of-the-art deep learning methods with DNNs, CNNs and RNNs on activity recognition problems, especially various LSTMs.

Statistical features. Kernel methods have been well studied during the past decades, with the ability to learn nonlinear transformations of input data as implicit features, and of learning nonlinear classifiers as well [Smola et al., 2007]. Recently, Muandet et al. [2017] illustrated the power of feature embedding on image classification, and Qian et al. [2018] investigated the similar technique on wearable-sensor-based activity recognition, with more reasonable and meaningful explanations on the extracted features. Similar idea has also been applied to the generative adversarial networks (GANs) with a different motivation of matching statistical features to enable the network to generate more realistic synthetic samples [Li et al., 2015; Li et al., 2017].

3 Preliminaries

The idea of extracting infinite number of statistical features may sound counter-intuitive, thus here we briefly introduce the kernel mean embedding technique [Smola et al., 2007; Muandet et al., 2017; Qian et al., 2018].

Given a sample X = {xi}ni=1 drawn from a probability distribution P, where X represents a set consisting of n elements xi ∈ Rd×1 . The technique of kernel embedding for representing an arbitrary distribution is to introduce a mean map operation µ(·) to map instances to a RKHS (Reproducing Kernel Hilbert Space) H, and to compute their mean in the RKHS as µP := µ(P) = Ex∼P[φk(x)] = Ex∼P[k(x, ·)], where φk : Rd → H is defined by a kernel k(·, ·). Here φk is a feature mapping function that extracts high-dimensional or even infinite-dimensional features from d-dimensional data space to Hilbert space H. If the condition Ex∼P(k(x, x)) < ∞ is satisfied, then µP is also an element in H. It has been proven that if the kernel k(·, ·) is characteristic, then the mapping µ : P → H is injective [Sriperumbudur et al., 2009]. The injectivity indicates an arbitrary probability distribution P is uniquely represented by an element in a RKHS through the mean map. In practice, one can use an unbiased empirical estimation to approximate the mean map ˆµP = 1n ∑ni=1 φk(xi) = 1n ∑ni=1 k(xi, ·). Though in theory, the dimension of ˆµP is potentially infinite, by using the kernel trick, the inner product of two probability distributions in a RKHS can be computed efficiently through a kernel function associated to the RKHS, 〈ˆµPx , ˆµPz 〉 = ˜ k(ˆµPx , ˆµPz ) = 1 nxnz ∑nx i=1 ∑nz j=1 k(xi, zj ), where ˜ k(·, ·) is a linear kernel defined in the RKHS, nx and nz are the sizes of the samples X and Z drawn from Px and Pz, respectively. In general, ˜ k(·, ·) can be a nonlinear kernel defined as ˜ k(ˆµPx , ˆµPz ) = 〈ψ(ˆµPx ), ψ(ˆµPz )〉, where ψ(·) is the associated feature mapping of the nonlinear kernel ˜ k(·, ·). These equations indicate that kernels can be applied on top of kernels. It is also possible to extract explicit low-dimensional feature maps z to substitute for the above implicit features by Random Fourier Features [Rahimi and Recht, 2007]: k(x, x′ ) = 〈φ(x), φ(x′ )〉 ≈ z(x)⊤z(x′ ), where the inner product of explicit feature maps can uniformly approximate the kernel values without the kernel trick.

4 The Proposed DDNNModel

4.1 The Overall Model

Activity recognition is challenging as it is affected by many factors, i.e., dynamic spatial-temporal correlations and varying patterns of activities conducted by multiple participants. Based on the above motivation, we design an end-to-end trainable neural network structure for human activity recognition problem. Our proposed model has three main modules to learn feature representations for human activity recognition:

• Statistical module f1: this module aims to learn all orders of moments statistics as features in an automated fashion.
• Spatial module f2: this module aims to learn correlations among sensors placements.
• Temporal module f3: this module aims to learn temporal sequence dependencies along the time scale.

By stacking the above learned features together and forming a unified architecture, we can build a trainable model for activity recognition. The overall illustration of the proposed model is shown in Figure 1.

In our problem setting of activity recognition, the streams of multivariate sensor readings are partitioned by fixedsize sliding window with length L. We randomly split activities into training set {(Xi, yi)}ni=1 , validation set {(Xj, yj)}mj=1 and test set {Xt}pt=1 , where each activity Xi = [xi1 ... xiL] = [x1i ... xdi ]T ∈ Rd×L, and yi ∈ {1 ... nc} with nc denoting the number of predefined activity categories. Here each column xij ∈ Rd×1 is a vector of signals received from d sensors at j-th timestamp, and each row (xri )T ∈ R1×L represents the signals recorded by r-th sensor within the current sliding window.

Note that in this work, we simply concatenate these three modules' learned features [f1(Xi), f2(Xi), f3(Xi)] before feeding into fully-connected layers. However, it is possible to explore more complex and interleaved ways to connect these modules depending on different scenarios. For instance, one possible choice is f1([f2(X), f3(Xi)]), with which statistical features are learned on top of the features extracted by other two modules. This is actually a generalized way of learning features, i.e., [f2(X), f3(X)] is considered as a special type of data transformation of raw data Xi, while f1(Xi) learns features directly from raw data. It is also possible to build a deeper model with these three modules as atom building blocks.

4.2 Statistical Module

Inspired by SMMAR [Qian et al., 2018], we aim to learn statistical features automatically by a deep learning model. One disadvantage of SMMAR is that the learned features are limited by a fixed Gaussian kernel k(x, x′ ) = exp(-γ‖x-x′ ‖2 ) with fixed γ, hence parameter tuning of proper bandwidth for kernel is required in advance. Here we aim to learn statistical features from multiple kernels without manual parameter tuning. This statistical module can be seamlessly combined with other modules to form a unified deep learning architecture which can be trained and optimized.

First, we aim to design a neural network f1 to learn the statistical feature mapping φf1 (·) automatically, i.e.,

However, the desired φf1 takes the matrix as input, while φk works for vectorial input. To address this issue, we take the average of feature mapping within each sliding window as

Second, we expect f1 is able to learn the best kernel automatically from different possible characteristic kernels k ∈ K.

Note that the learned features f ∗ 1 (Xi) are in vectorial form. As mentioned in Section 3, the prerequisite of expressive feature extraction is the characteristic property of kernels, i.e., the feature mapping f1(·) should be injective (not necessarily invertible). To make the neural network injective, there should be another function or neural network f -1 1 such that f -1 1 (f1(Xi)) = Xi for all possible Xi's. Therefore, as suggested in [Li et al., 2017], we utilize an autoencoder to guarantee the injectivity of the feature mapping.

To be specific, an autoencoder includes an encoder fe, and a decoder fd, where the encoder is used to map the input sequence to a fixed-length vector, then the decoder is used to unroll this vector to sequential outputs and try to reconstruct the input data of the encoder. In our scenario, the encoder is the desired f1 module, and fd = f -1 1 . Though both of our proposed model and the model in [Li et al., 2017] utilize an autoencoder to make sure the injectivity of neural networks, the motivations are quite different. We utilize the autoencoder as feature learner for classifying activity classes, while in their model, the autoencoder works for hypothesis testing, i.e., to make generated synthetic samples as indistinguishable from true samples as possible.

The standard loss function of the autoencoder tries to minimize the reconstruction error ℓae = ‖x-fd(fe(˜x))‖ between inputs x and outputs ˜x, but it is insufficient for statistical feature learning. We further use an extra loss function based on MMD distance to force the autoencoder to learn good feature representations of inputs:

where np and nq are the numbers of timestamps of two activities Xp and Xq, respectively. The resultant MMD loss function on the autoencoder is as follows:

Note that by taking fe and fd to be the identity function, ℓMMD is reduced to ℓae, where the mean vector (1st order moment) difference between inputs and outputs of the autoencoder is calculated. Our choices for fe and fd in the proposed deep learning model aim to match higher order moments statistical features. Therefore, this loss function forces the hidden representations of autoencoder to successfully convey sufficient information of desired statistics to the decoder.

4.3 Spatial Module

Convolutional layers in CNNs are firstly designed for the image-based problems. The standard CNNs are able to extract spatial-invariant features with a kernel filter running over the images or videos. However, the current so-called CNNs for human activity recognition tasks are actually not truly on spatial dependencies. Usually the wearable sensor data is in 1-dimension, where the so-called spatial CNNs are actually along the temporal aspect [Hammerla et al., 2016; Morales and Roggen, 2016]. There are also attempts to force the multiple 1-dimensional data of different sensor channels into a virtual image, then standard CNNs can be applied [Yang et al., 2015].

Our viewpoint of spatial correlations are different from the previous work. We try to capture the spatial correlations between sensors attached to the human body. From our point of view, the signals of a certain sensor are inevitably affected by the attached locations on the human body or joints. Imagine a participant is walking, with sensors attached to his upper arm, lower arm and legs. It is common that when the right leg of the participant is on the front, the right arms are waved to the opposite direction at the same time. Also the movements of upper arm and lower arm are constrained by the joints of the human body. Therefore we aim to model such kinds of spatial correlations of the sensors, which is usually ignored in the literature. As illustrated in Figure 1, the input data Xi in the sliding window is treated as d row vectors [x1i ... xdi ]T , each of which associated to a single sensor. A LSTM is connected with each sensor data, and hence the dependencies between sensors are learned to form a spatial feature vector.

4.4 Temporal Module

In order to exploit the temporal dependencies within each activity, we utilize both CNNs and LSTMs as building blocks of temporal module. As discussed in previous subsection, CNNs with 1-D filters are applied on each channel {xri }dr=1 of sensor data Xi. By applying the filter to go through different regions of the input, it is then able to detect the local salience patterns of the signals. Note that CNNs are applied along the temporal dimension, thus it is able to learn the temporal dependencies. Besides, LSTMs are connected to temporal data {xij}Lj=1 to learn temporal information as well. Specifically, we choose LSTMs instead of RNNs due to the diminishing gradients problem. LSTMs are designed to have more dynamic and flexible memory cells through gating mechanism, which enables LSTMs to learn temporal relationships on longer time scales. The outputs of CNNs and LSTMs are concatenated into a single vector to represent temporal features.

5 Experiments

Dataset. We conduct experiments on four sensor-based activity datasets. The overall statistics information of datasets are listed in Table 1. The Daphnet Gait dataset (DG) [B¨achlin et al., 2010] corresponds to a medical application and records activities from 10 participants affected with Parkinson's Disease, aiming to detect freezing of gait incidents. The data is segmented by sliding window of 1 second duration and 50% overlap. The Opportunity dataset (OPPOR) [Chavarriaga et al., 2013] comprises 17 mid-level gesture classes conducted in an ambient-sensor home environment together with 19 onbody sensors. These gestures are short in duration and nonrepetitive. Null class data exists in the dataset indicating transitions of two adjacent activities. The UCIHAR dataset [Anguita et al., 2012] collects six activities (walking, walking upstairs, walking downstairs, sitting, standing, laying) carried out with a group of 30 volunteers within an age range of 19-48 years. The PAMAP2 dataset [Reiss and Stricker, 2012] includes 12 different physical activities (household activities and exercise activities) which are performed by 9 subjects wearing 3 inertial measurement units. These activities are prolonged and repetitive, typical for systems aiming to characterize energy expenditure.

5.1 Experimental Setup

All these datasets have class imbalance problem, especially OPPOR and DG. Therefore, in our experiments, we set the probability of an activity being chosen in a training epoch to be the inverse of the number of the certain activity. We follow the experimental setup in [Hammerla et al., 2016]. Micro-F1 (miF) and weighted macro-F1 (maF) are selected as performance measure. 77 out of 113 features are used for OPPOR, with run 2 from subject 1 as validation set, runs 4 and 5 from subject 2 and 3 as test set and the rest as training set. Sliding windows of 1 second duration with 50% overlap is applied. For PAMAP2, 12 protocol activities are studied, with data downsampled to 33Hz. Sliding window length is 5.12 seconds with 1 second as step size. Runs 1 and 2 for subject 5 are used as validation set, and runs 1 and 2 for subject 6 are used as test set, with the rest being training set. The raw data of DG is downsampled to 32Hz as well. Sliding window duration is 1 second with half overlap. We use subject 9's first run as validation set, subject 2's runs 1 and 2 as test set with the rest being training set. The UCIHAR has been preprocessed and segmented by data provider beforehand, where the raw data is randomly partitioned into two sets, where 70% of the volunteers generated training data and 30% the test data. The sensor signals were pre-processed by applying noise filters and then sampled in sliding windows of 2.56 second (128 readings). Data normalization is conducted on all datasets. For our architecture, we utilize 4 linear layers with ReLU attached after each linear layer as encoder and decoder's architecture. Both LSTMs in spatial and temporal module have l layers of LSTMs with h-dimensional hidden representations, where l ∈ {1, 2, 3} and h ∈ {32, 64, 128, 256, 512, 1024}. Four convolutional layers with filter size (1, 5) are utilized in the temporal module, with ReLUs and max pooling layers attached after each convolutional layer. All feature vectors are concatenated into a single vector before feeding into three fully-connected layers. The batch size is set to 64, and the maximum training epoch is 100. Adam optimizer is used for training with learning rate 10-3 and weight decay 10-3 . All experiments are run on a Tesla V100 GPU. 1

Baselines. We compare our proposed model with baseline methods as well as state-of-the-art methods. Due to the fact that feature-engineering-based machine learning methods are hard to scale up, in this paper we mainly compare our proposed DDNN model with deep learning based methods.

1 Code of the proposed DDNN is available at https://github.com/ Hangwei12358/IJCAI2019 DDNN.

• DDNN-f1: the proposed deep model without the statistical module. This baseline is set to investigate the efficacy of the statistical module.
• DDNN-f2: the proposed deep model without the spatial module. This baseline is set to investigate the efficacy of the spatial module.
• CNN Yang [Yang et al., 2015]: a state-of-the-art CNNbased model with 3 convolutional layers. We follow the architecture in the paper and reproduce the model.
• DeepConvLSTM [Morales and Roggen, 2016]: a stateof-the-art model with 4 convolutional layers and 2 LSTM layers. We also follow the architecture and reproduce the model.
• DNN: 5-layer linear transformation with ReLU activation function.
• CNN: 4-layer CNNs with kernel size (1, 5) with ReLU activation function and max pooling layer attached to the output of each CNN.
• LSTM: 2-layer LSTMs with the dimension of hidden representation in the range {32, 64, 128, 256, 512}.
• LSTM-f, LSTM-S, b-LSTM-S: state-of-the-art LSTMs variants to capture temporal sequences information. Results are directly from [Morales and Roggen, 2016].

5.2 Experimental Results and Analysis

The results of the proposed method and baselines on 4 datasets are listed in Table 2. The best performance for each evaluation metric is highlighted in bold. Our proposed DDNN has achieved the best performance on all datasets, except for the maF of OPPOR. These results indicate that our proposed model is capable of learning powerful various features for classification of activity recognition with more discriminative power.

Impact of spatial and statistical module. Remarkably, the performances of DDNN are consistently better than those of DDNN-f1 and DDNN-f2 on all datasets. This favorably validates our motivation that statistical features and spatial features are beneficial to the deep learning models besides the widely used temporal features in existing literature.

Robustness of the proposed DDNN. One interesting finding is that our proposed model is more robust than other baselines. For instance, LSTM-related methods are obviously inferior on DG and UCIHAR, and CNN-based models are much worse than other baselines in OPPOR. One possible reason may lie in the unified framework of DDNN, where different aspects of features are learned together. It is reasonable that the contributions of different features on the classification performance are task-dependent, i.e., the importance of statistical module f1 and spatial module f2 varies in datasets since each dataset has unique characteristics on properties.

Parameters' sensitivity. Another aspect of robustness is found during parameter tuning, where DDNN is less sensitive to the changes of parameters. For example, when we set the number of LSTM layers to be {1, 2, 3}, and LSTMs hidden representation dimensions to be {32, 64, 128, 256, 512}, the performance difference of DDNN is only roughly several percentage, while other models' performance gap is larger. We also investigate the dimensions of hidden representations in the autoencoder of statistical module ranging from 0.5d to 10d with d indicating the number of dimensions of raw data. Empirically, higher dimensional hidden representations actually hinder the performance of deep model, while the dimensions lower than 4d does not affect the performance drastically. We also investigate the weights on the added loss function ℓMMD for statistical module. We conduct experiments with various weights put on the loss function. As illustrated in Figure 2, the performance is steady (ranging from 0.88 to 0.9) within the weight ranging from 10-4 to 101 , but when the weights are larger than 101 , the performance degrades drastically. The reason may be the large weights on the ℓMMD leads to less contribution of the rest two modules (temporal and spatial), which affects the final performance.

6 Conclusion and FutureWork

In this paper, we propose a novel architecture for wearablesensor-based activity recognition tasks. Our proposed DDNN model is able to automatically learn three types of features: 1) statistical features, 2) spatial correlations among sensors, and 3) temporal features. Extensive experiments with analysis are conducted to compare with state-of-the-art methods. Experimental results demonstrate the superior efficacy of the proposed model. In the future, we plan to extend DDNN to semi-supervised setting where the number of labeled training data is limited.

Acknowledgments

This research is partially supported by the Singapore Ministry of Health under its National Innovation Challenge on Active and Confident Ageing (NIC Project No. MOH/NIC/HAIG03/2017). Sinno J. Pan thanks the support from the NTU Singapore Nanyang Assistant Professorship (NAP) grant M4081532.020, and Singapore MOE Tier-1 grant 2018-T1-002-143.