Multivariate Times Series Classification Using Multichannel CNN

1 Introduction

Multivariate time series classification (MTSC) has attracted considerable attention in the literature because it is widely applied in diverse areas such as health monitoring, motion recognition, gas mixture classification, and bioinformatics. Consequently, numerous algorithms have been developed for this task [Bagnall et al., 2018; Ismail Fawaz et al., 2019]. In this study, we focused on deep learning-based MTSC models that utilize convolutional neural networks (CNNs). Oh and Kim [2021] introduced multichannel representation of multivariate time series and its corresponding CNN architecture for gas mixture classification. Their proposed approach transforms multivariate time series into multichannel analogous images and utilizes well-established computer vision models pretrained by a large dataset, e.g. ImageNet. In this study, we expand the multichannel CNN classifier to a different dataset and compare it with recent deep learning methods for MTSC.

2 Problem Statement

Suppose that we observe N classes belonging to the given dataset C = {c1, . . . , cnc } as follows:

where nc denotes the number of classes. Let Xi = (xmt) ∈ RM×T denote a multivariate time series, where T is the number of time steps and M is the number of features. To predict the classes of Xi, we must construct a classifier F:

F can be trained by minimizing the discrepancy between the predicted class ˆ yi and the ground-truth class yi. Here, we adopt a positive loss function L to reflect this discrepancy.

3 Methodology

In general, Xi is not an appropriate form for 2D CNN model. Thus, we transform the original time series into an analogous image matrix before applying a CNN. The core concept is to transform the original time series Xi into an analog image matrix, as described by Oh and Kim [2021]. It generates M grayscale images using a preprocessing function. For example, the RacketSports dataset consists of six features that can be transformed into a six-dimensional analogous image matrix, as shown in Figure 1.

After converting the multivariate time series into an image, the ResNet50 model is modified with a multichannel input. The input multivariate time series data are then converted into an analog image matrix and passed through corresponding channels. Subsequently, we assign individual time series to individual channels and apply a convolutional operation. In the case of the RacketSports dataset, the number of input channels is the same as the number of features, that is, six (= M). The final fully connected layer has four (= N) nodes for each class. In addition, we use the ImageNet pretrained weight for transfer learning.

In previous studies, CNN models have shown promising results for MTSC [Ismail Fawaz et al., 2019]. Given an input multivariate time series, a convolutional layer consists of sliding filters over the time series, thereby enabling the network to extract nonlinear discriminant features that are time invariant and useful for classification. Thus, the proposed multichannel CNN approach enables the extraction of the nonlinear characteristics of multivariate time series.

4 Experiments

We compared five conventional deep learning-based time series classification methods to the proposed method. Wang et al. [2017] proposed three baseline models: multi-layer perceptron (MLP), fully convolutional neural network (FCN), and residual neural network (ResNet). The time-CNN approach proposed by Zhao et al. [2017] comprises two consecutive convolutional layers with a sigmoid activation function. Ismail Fawaz et al. [2020] introduced InceptionTime, which is inspired by the Inception architecture. Benchmark methods were implemented using the Python libraries sktime-dl 1 .

We validated the proposed approach using datasets from the University of East Anglia (UEA) multivariate time series classification database [Bagnall et al., 2018]. We utilized five different MTSC datasets with small amounts of training data, as presented in Table 2: Epilepsy (EP), EthanolConcentration (EC), RacketSports (RS), StandWalkJump (SWJ), and UWaveGestureLibrary (UWG).

We compared the weighted F1 score for five-fold experiments and the average rank for each problem. In Table 1, the average of the five-fold scores is shown, and the ranks are compared. Proposed (False) is ResNet50 with random initialization, and Proposed (True) is ResNet50 with the ImageNet pretrained model. The proposed method with transfer learning is superior on EP and EC problems but underperforms on RS, SWJ, and UWG data. On average, the proposed approach achieved the highest ranking. As expected, transfer learning improved the classification performance, even with a pretrained weight from a totally different task.

1 https://github.com/sktime/sktime-dl

5 Conclusion

We propose image transformation and CNN approach for MTSC. The proposed method exhibited better classification performance than previous approaches on the selected five datasets. Hence, it can be applied to different MTSC datasets. However, the proposed approach was validated using only five multivariate time series. Thus, future work will involve conducting more experiments on different MTSC datasets. Also, the proposed approach requires computation of the transformed image. The computational costs and inference times should be investigated in future studies.

Acknowledgments

This research paper is a part of my doctoral dissertation. I would like to express my heartfelt gratitude to Prof. Sungil Kim, my Ph.D. advisor, for his unwavering support, patience, encouragement, passion, and vast knowledge.