Time-Aware Multi-Scale RNNs for Time Series Modeling

1 Introduction

Time series is a set of values sequentially recorded over time. Modeling of time series can provide meaningful knowledge, which is beneficial for decision-making in a variety of fields, such as human motion prediction [Martinez et al., 2017] and EEG/ECG data analysis [Bagnall et al., 2018].

Multi-scale information is crucial to the modeling of time series [Mozer, 1992; Koutnik et al., 2014]. Currently, most methods use multi-scale convolution [Cui et al., 2016] or skip connections to model multiple scales of time series [Koutnik et al., 2014; Chang et al., 2017; Chiu et al., 2019; Carta et al., 2020]. Multi-scale convolution uses different downsampling frequencies or convolution kernels of different sizes, while models with skip connections capture multiple scales through direct connections spanning different lengths. These methods use pre-fixed multiple scales, assuming that all the scales are equally important for each sample.

∗Qianli Ma is the corresponding author.

However, it is difficult for these fixed-scale methods to capture the dynamic temporal patterns of time series. For example, Fig. 1 shows two music clips of different genres. "Hip-Hop" is a style of music with a strong beat, while "Experimental" is relatively chaotic. Intuitively, "Hip-Hop" requires a larger scale to capture long-term dependencies due to its regularity, while "Experimental" needs a smaller scale to capture short-term dependencies due to its sharp fluctuations. Hence, we need to adaptively select a suitable scale for each sample rather than a fixed one. Meanwhile, the recognition of genres requires modeling the emotional changes in music, which are controlled by note duration. Therefore, different scales are also needed at different time steps as the notes have different durations at different times [Hu et al., 2019].

Recently, some methods have been proposed to select appropriate scales corresponding to each sample dynamically [Neil et al., 2016; Jernite et al., 2016; Campos et al., 2018; Hu et al., 2019]. These methods select a specific scale adaptively or decide whether to update the hidden state at each time step of the Recurrent Neural Network (RNN). However, these methods attempt to use uniform feature representations to model multiple scales, entangling representations of different scales. It is hard to explore the impact of different scales using the entangled representation, which is harmful to the interpretability of the model [Tamkin et al., 2020]. Hence, disentangling the scales helps learning the representations of different scales better and is conducive to interpretability.

To address the above issues, we propose Time-Aware Multi-Scale RNNs (TAMS-RNNs) that can adaptively model the multi-scale information for each time series at each time step. First, we design a mechanism called Multi-Scale Feature Disentanglement (MSFD) to obtain decoupled feature representations of different scales. Concretely, the hidden state of the RNN is disentangled into multiple independently updated small hidden states, which use different update frequencies to model the multi-scale information. Furthermore, a mechanism called Time-Aware Feature Modulation (TAFM) is designed to modulate the features of different scales, adaptively selecting the most important scale at each time step. We conduct comparative experiments on Multivariate Time Series (MTS) classification, human motion prediction, and music genre recognition to verify the superiority of TAMS-RNNs. The contributions of our work are:

• We propose Time-Aware Multi-Scale RNNs (TAMSRNNs) to adaptively model the multi-scale information for time series modeling. It uses a Multi-Scale Feature Disentanglement (MSFD) mechanism to disentangle feature representations of different scales.
• We design a simple but effective mechanism we call Time-Aware Feature Modulation (TAFM) that uses temporal context information to modulate the features of different scales, adaptively selecting the most important scale for each sample at each time step.
• We conduct extensive experiments showing that our model outperforms state-of-the-art methods on MTS classification and human motion prediction tasks. Furthermore, we visualize the network's behavior on music genre recognition, verifying the model's effectiveness.

2 RelatedWork

Many methods have been proposed that use RNNs to model multi-scale time series dynamics. They can be roughly divided into two categories:

Pre-fixedmulti-scale. ClockworkRNN (CW-RNN) [Koutnik et al., 2014] divides one layer of the RNN into separate modules, each processing inputs at its own temporal granularity. Dilated RNN [Chang et al., 2017] extracts the representations of multiple scales by stacking multiple layers of RNNs and using skip connections of different lengths. MS-LMN [Carta et al., 2020] separates the RNN into different modules with different sampling rates, using an incremental training algorithm to learn long-term dependencies. However, these methods use fixed, predefined scales, making it difficult to adapt to different time series dynamics.

Learnable multi-scale. There are also methods that select suitable scales dynamically. Phased LSTM [Neil et al., 2016] and Skip RNN [Campos et al., 2018] decide whether to update the hidden state at each time step of the RNN to learn representations at multiple scales. VCRNN [Jernite et al., 2016] adaptively determines the number of neurons to be updated at each time step. ASRNN [Hu et al., 2019] presets multiple scales and selects a specific scale at each time step. However, these methods attempt to use uniform feature representations to model multiple scales, and the information of various scales are entangled together. Hence, it is difficult to explore the impact of different scales on the model.

The most relevant model to ours is CW-RNN [Koutnik et al., 2014]. However, in CW-RNN, representations of different scales are entangled together, and it is difficult for the model to capture the dynamic temporal patterns of time series. We update the small hidden states independently to learn the representations of different scales better. Meanwhile, the temporal context information is used to select the most important scale at each time step adaptively, capturing more complicated temporal patterns. A detailed comparison of their update process to ours is shown in Fig. 3.

3 Time-Aware Multi-Scale RNNs

We propose Time-Aware Multi-Scale RNNs (TAMS-RNNs) and design two mechanisms (named Multi-Scale Feature Disentanglement and Time-Aware Feature Modulation, respectively) for TAMS-RNNs to adaptively model the multi-scale information for time series modeling. The whole architecture of TAMS-RNNs is shown in Fig. 2.

3.1 Multi-Scale Feature Disentanglement

Given time series X = {x1, x2, · · · , xT } where xt ∈ Rd0 , d0 is the dimension of input data and T denotes the length of time series, the hidden state ht ∈ Rd of RNN cell can be expressed as follows:

where xt and ht are the input and hidden state at time step t, respectively. W ∈ Rd×d0 , U ∈ Rd×d , and b ∈ Rd are the learnable parameters. f denotes the hyperbolic tangent activation function tanh.

To capture the feature representations of different scales, the mechanism of MSFD is designed. First, the hidden state of RNN is disentangled into multiple small hidden states that are updated independently. Specifically, the hidden state at time step t is disentangled into K small hidden states as ht = [h1t , · · · , hKt ], where hkt ∈ Rp , p = d/K and [·] is the concatenation operation. The corresponding recurrent matrix is defined as ˜U = [ ˜U1 , · · · , ˜UK ], where ˜Uk ∈ Rp×p . Each small hidden state hkt is independently updated by an individual recurrent matrix ˜Uk and then merged via concatenation to constitute the hidden state ht. Meanwhile, The learnable parameter W is defined as W = [W1 , · · · ,WK ], where Wk ∈ Rp×d0 . The update equation of ht is defined as:

where f denotes the hyperbolic tangent activation function tanh and ˜b ∈ Rp is the learnable parameter. Specifically, the model is equivalent to RNNwhen K = 1.

After the disentangling process, each small hidden state would use a specific update frequency to capture information of a particular scale. Suppose the scale set S is {s1, · · · , sK}, for simplicity, sk is usually set to a power of 2 in our experimentation. For scale sk, the small hidden state hkt will be updated every sk time steps. As shown in Fig. 2, the entire hidden state of RNN is disentangled into 3 small hidden states, and the corresponding scale set is {1, 2, 4}. Scale s1

means the small hidden state is updated every time step, while the hidden state of scale s3 is updated every four time steps. Specifically, the update equation of hkt is defined as follow:

When t MOD sk ̸= 0, the "COPY" operation is used, directly copying the hidden state at the last time step. Otherwise, the "UPDATE" operation is employed, using the equation (2) to update the hidden state. After disentangling the hidden state with different update frequencies, each small hidden state is able to capture information of a certain scale. The hidden state with a small scale captures the short-term dependencies, which is crucial for the time series where the values change frequently. Meanwhile, the hidden state with a large scale captures the long-term dependencies, which provides essential information to time series modeling tasks.

Furthermore, we compare the update process of the proposed TAMS-RNN with RNN and CW-RNN [Koutnik et al., 2014], which is shown in Fig. 3. Firstly, compared to RNN, CW-RNN and TAMS-RNN separate the hidden state of RNN into multiple small hidden states, using different update frequencies to capture the multi-scale information of time series. However, in CW-RNN, representations of different scales are entangled together. For example, the hidden state of scale s2

will affect the update process of the small-scale hidden state. Compared to CW-RNN, TAMS-RNN updates the small hidden states independently to learn the representations of corresponding scales better and is also conducive to the interpretability of the model [Tamkin et al., 2020].

3.2 Time-Aware Feature Modulation

After the proposed MSFD, the model still uses multiple fixed scales, which is difficult to capture the dynamic temporal patterns of time series. For example, as shown in Fig. 1, the temporal patterns of two music clips are quite different, and even the patterns of different time steps within the same music series are also different. Therefore, different temporal scales are needed to model the corresponding patterns. To solve the problem, we design a simple but effective mechanism named TAFM that uses temporal context information to modulate the features of different scales so that the model can focus on different scales at different time steps.

First, the temporal context information ht-1 and xt are utilized to generate the distribution αt for each time step:

where αt ∈ RK indicates the importance degree of K small hidden states, and the value of αk t is in the range of 0 to 1. W′ ∈ RK×d0 , U′ ∈ RK×d and b′ ∈ RK are learnable parameters. A larger αk t means that the small hidden state of scale sk is more important for current time step. Then, αt is utilized to modulate the features of different scales, and the update equation of hkt is changed to:

Specially, when the "COPY" operation is used, hkt-1 is not multiplied by αk t , because continuously multiplying by values less than 1 will make the value of feature close to zero, which will lead to gradient vanishing problem. Thus, αk t is utilized to weight hkt-1 only when the "UPDATE" operation is used. After modulating the features of different scales, the model can adaptively select the most important scale at each time step and capture more complex dynamic temporal patterns.

Since our model is based on vanilla RNN, the recurrent cell of TAMS-RNN can be easily replaced with Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), which we refer to as TAMS-LSTM and TAMS-GRU.

4 Experiments

Experiments are conducted on MTS classification, human motion prediction and music genre recognition to verify the superiority of the model.

4.1 Multivariate Time Series Classification

Following TapNet [Zhang et al., 2020], we conduct experiments on 15 data sets from the latest MTS classification archive [Bagnall et al., 2018]. These data sets come from various fields with different numbers of classes and variables. Classification accuracy is computed as the evaluation metric.

For MTS classification, following previous work, we use LSTM as the recurrent cell of our model. The number of layers of TAMS-LSTM is set to 2, the hidden state dimension is set to 256 (d = 256), and the hidden state of the final time step is used for classification. Meanwhile, the number of small hidden states is set to 4(K = 4) with the scale set {1, 2, 4, 8}. We apply the dropout operation [Srivastava et al., 2014] to the input time series X with dropout rate of 0.1. The gradient-based optimizer Adam [Kingma and Ba, 2014] is chosen, and the learning rate is set to be 0.001.

We compare the proposed model with six benchmark methods, including deep learning methods (MLSTM-FCN [Karim et al., 2019] and TapNet [Zhang et al., 2020]), bag-of-patterns method (WEASEL+MUSE [Sch¨afer and Leser, 2017]), and common distance-based methods (ED/DTW-NN [ShokoohiYekta et al., 2015]). To fairly compare TAMS-LSTM with CW-RNN [Koutnik et al., 2014], we design a baseline model called "CW-LSTM", replacing the RNN cell of CW-RNN with LSTM. The experiment result is shown in Table 1. The result "N/A" in the table indicates the corresponding approach is unable to run because of computational issues. Meanwhile, the best rate of each model is computed and shown to better describe the performance of these models. The performance comparison shows that TAMS-LSTM outperforms other methods on 13 out of 15 MTS classification data sets and yields the highest best rate of 0.867, which is significantly better than the existing state-of-the-art approach TapNet with the best rate of 0.267. In addition, TAMS-LSTM is also significantly better than CW-LSTM, which illustrates the effectiveness of MSFD and TAFM.

4.2 HumanMotion Prediction

In addition to these 15 MTS data sets, we also conduct experiments on human motion prediction, using the Human 3.6M (H3.6M) data set [Ionescu et al., 2013]. H3.6M is one of the largest publicly available data sets of human motion capture, including seven actors performing 15 activities. Following previous work [Jain et al., 2016; Martinez et al., 2017], we use six actors for training and the actor five for testing. Meanwhile, we train our model for long-term prediction, forecasting the future 25 frames (1000ms). The error is measured by the euclidean distance between the predicted sequences and ground truth, in terms of the Euler angles.

For human motion prediction, RRNN [Martinez et al., 2017] is first proposed, using a sequence-to-sequence architecture with residual connections, which results in much more accurate prediction results. Therefore, under the architecture of RRNN, we replace the vanilla GRU in RRNN with the proposed TAMS-GRU (refer to as TAMS-RRNN) to model the multi-scale information existing in skeleton-based action sequences, such as movements of different amplitudes. Meanwhile, we also compare the proposed model with LSTM3LR [Fragkiadaki et al., 2015] and cgRNN [Wolter and Yao, 2018]. Finally, since TP-RNN [Chiu et al., 2019] is a recently proposed advanced model based on recurrent architecture, we also replace the vanilla LSTM in TP-RNN with the proposed TAMS-LSTM (refer to as TAMS-TP-RNN) to further verify the effectiveness of the proposed model. Since RRNN [Martinez et al., 2017] and cgRNN [Wolter and Yao, 2018] are only trained for short-term prediction (up to 400ms), we train their models for long-term prediction (up to 1000ms) using the publicly available implementations 1 . In TAMS-RRNN and TAMS-TP-RNN, the number of small hidden states is set to 4(K = 4) with the scale set {1, 2, 4, 8}. Other configurations are consistent with the corresponding papers.

The experiment result is shown in Table 2. After replacing the vanilla RNN cell with the proposed TAMSRNN cell, TAMS-RRNN is significantly better than RRNN, while TAMS-TP-RNN also achieves better or competitive results compared with TP-RNN. The results show that TAMSRNN is capable of capturing the multi-scale information of skeleton-based action sequences. The detailed results of TPRNN and TAMS-TP-RNN are shown in Appendix A 2 .

4.3 Music Genre Recognition

Furthermore, since multi-scale structure naturally exists in music time series [Hu et al., 2019], we conduct experiments on music genre recognition to verify the effectiveness of the proposed TAMS-RNN. Music genre recognition is a challenging task because the boundaries between different genres are hard to distinguish due to people's subjective feelings. Following ASRNN [Hu et al., 2019], we choose the FMAsmall data set [Defferrard et al., 2016] to conduct our experiments, which contains 8000 music clips of 8 genres. We follow the standard 80/10/10% data splitting protocols to get training, validation and testing sets and directly employ raw music clips as inputs. The sampling rate is reduced to 200 Hz, resulting in very long sequences with about 6000 time steps.

1 https://github.com/una-dinosauria/human-motion-prediction, https://github.com/v0lta/Complex-gated-recurrent-neural-networks 2 https://github.com/qianlima-lab/TAMS-RNNs

For music genre recognition, the one-layer TAMS-LSTM is chosen, and the hidden state of the final time step is used for classification. Meanwhile, the number of small hidden states is set to 4(K = 4) with a large scale set {1, 4, 16, 64} due to the long input sequences. The Adam optimizer [Kingma and Ba, 2014] is used, and the learning rate is set to 0.001.

We compare TAMS-LSTM with several recurrent models, including SLSTM, ASLSTM [Hu et al., 2019], and LSTM. To keep the number of parameters consistent with ASLSTM, the hidden state dimension of TAMS-LSTM is set to 192 (d = 192). The recognition accuracy is computed as the evaluation metric, and the result is shown in Table 3. The performance comparison shows that TAMS-LSTM is significantly better than ASLSTM, verifying the effectiveness of the model.

Furthermore, for each genre, we count the weights assigned to each scale, averaging over all the samples and all the time steps t when t MOD sK = 0 (sK = 64). The results are normalized and shown in Fig. 4. Firstly, all the genres prefer to choose larger scales (scale s3 and s4) since the input sequences are so long, and large scale is conducive to modeling the long-term dependence of time series. Meanwhile, there are also significant differences between music of different genres. "Hip-Hop" and "Electronic" prefer to choose the largest scale, compared with the other six. More specifically, Fig. 5 shows three music clips and the corresponding weights assigned to the scales. Music such as "Hip-Hop" is rhythmic and regular, which requires a large scale to model its regularity, thus the weight assigned to scale s4 is large. On the contrary, a smaller scale is sufficient for chaotic music "Folk", since the values of this time series are only related to values within a local window. Therefore, the model would assign similar weights to all four scales. Meanwhile, at each time step t, we record the scale with the largest weight, which is shown in Figure 5(f). The music series in the yellow region has small changes, so there is no need for the model to be updated frequently, and a larger scale would be chosen. On the contrary, series in the purple region varies greatly, thus the choice of scale is more flexible. Similar observations for more examples are shown in the Appendix B. In general, our model better models the multi-scale information of time series and improves the interpretability of the model.

4.4 Ablation Study

Ablation study is conducted on 2 MTS data sets, and the results are shown in Table 4. "LSTM+MSFD(M)" outperforms "LSTM+MSFD(S)" and "LSTM", indicating the importance of multi-scale information. Meanwhile, "LSTM+MSFD(M)"

Time

Table 4: Ablation study on 2 MTS classification data sets. "S" means using single frequency to update different small hidden states, while "M" means using multiple frequencies.

outperforms "CW-LSTM", verifying the effectiveness of MSFD. Finally, the full model (last column) outperforms "LSTM+MSFD(M)" and "CW-LSTM+TAFM", which illustrates the effectiveness of TAFM and MSFD, respectively. The results of other data sets are shown in Appendix C.

4.5 Model Analysis

Impact of the value K. To study the influence of the value K (the number of small hidden states), we conduct experiments on 2 MTS classification data sets, FaceDetection and Phoneme. K is set in the range of 1 to 6, and the corresponding scale sets are set to {1}, {1, 2}, {1, 2, 4}, {1, 2, 4, 8}, {1, 2, 4, 8, 16} and {1, 2, 4, 8, 16, 32}, respectively. The results are shown in Figure 6(a). We found that the optimal value K is 4. Increasing K helps to model the multi-scale information of time series, while the hidden size is too small to provide enough features when K is too large.

Impact of the scale set S. The impact of the scale sets is explored and shown in Figure 6(b). 1 to 4 denotes the scale set {1, 2, 3, 4}, {1, 2, 4, 8}, {1, 4, 8, 16} and {1, 4, 16, 64}. The optimal one is {1, 2, 4, 8}. When the range of scales is small, the difference between scales is relatively small, making it hard to learn multi-scale information. When the range of scales is too large, it is hard for a large scale to learn enough information because most time steps are skipped. Other analysis of the model is shown in Appendix D.

5 Conclusion and FutureWork

In this paper, we propose Time-Aware Multi-Scale RNNs (TAMS-RNNs) to capture the multi-scale dynamics of time series adaptively. Instead of using pre-fixed multiple scales, our model disentangles representations of different scales and adaptively selects the most important scale for each sample at each time step. Our experiments demonstrate that TAMSRNNs outperform state-of-the-art methods on different classic time series tasks. Furthermore, the visualization analysis on music genre recognition verifies the effectiveness of the model. In TAMS-RNNs, we use a simple strategy (power of 2) to set the value of each scale, which may not be flexible enough. In future work, we plan to adaptively learn appropriate scale set and apply the model for more other tasks.

Acknowledgements

We thank Prof. Garrison W. Cottrell for his helpful comments and revisions of this paper. The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant Nos. 61502174, 61872148), the Natural Science Foundation of Guangdong Province (Grant Nos. 2017A030313355, 2017A030313358, 2019A1515010768, 2021A1515011496), the Guangzhou Science and Technology Planning Project (Grant Nos. 201704030051, 201902010020), the Key R&D Program of Guangdong Province (Grant No. 2018B010107002), and the Fundamental Research Funds for the Central Universities.