DeLELSTM: Decomposition-based Linear Explainable LSTM to Capture Instantaneous and Long-term Effects in Time Series

1 Introduction

Time series forecasting is ubiquitous across a broad range of applications, including finance [Wu et al., 2013], meteorology [Chakraborty et al., 2012], energy consumption [Wang et al., 2020], and medical health [Zhang, 2019]. Deep neural networks have been successfully developed for time series forecasting tasks. Among them, Recurrent Neural Networks (RNNs) and its variant, the Long-Short TermMemory (LSTM) [Hochreiter and Schmidhuber, 1997] and the Gated Recurrent Units (GRU) [Cho et al., 2014], are widely used networks for handling sequential data and have been proven to be powerful tools [Guo et al., 2019].

∗Corresponding author.

Despite the promising results of RNNs in time series forecasting, using RNNs to modelling time series lack interpretability, which is critical in high-stakes applications like finance and healthcare. For example, in the medical field, e.g., predicting the mortality rate after patients enter the ICU, it is crucial for clinicians to understand how models output a specific prediction and which indicators are useful. Such explanations can aid reliable decision-making for clinicians and increase trust in models' predictions.

Although much recent work has been done on explainability in the computer vision and natural language processing [Masoomi et al., 2021; Mohankumar et al., 2020; Tsang et al., 2020], this problem has been overlooked in the case of time series forecasting [Tonekaboni et al., 2020; Rojat et al., 2021; Hsieh et al., 2021]. The time series is special because of its dynamic nature, which causes multivariable patterns to change over time and makes it more difficult to build explainable models.

There are two main challenges in explaining time series forecasting models. First, each variable has a different impact on the target series, and the effects of variables on the target series are dynamic over time. Therefore, capturing the different dynamic impacts of each variable and distinguishing the contribution of each variable to the prediction is difficult for explaining the forecasting model. Second, the serial dependencies for each variable are heterogeneous. When predicting target series, the long-term effects of some variables play a decisive role, while the instantaneous effect of other variables is more important. For example, in the financial market, investors tend to utilize multivariate time series, such as the stock index and related stocks, to forecast the stock prices at the next time point. If the instantaneous influence and the long-term effects of each feature can be distinguished, investors can focus on more significant information and make appropriate investment decisions. As a result, we should solve the challenge of distinguishing the long-term effect and the instantaneous influence of each variable.

Existing explainable RNNs primarily use the attention mechanism on the hidden states to get the important variables [Tonekaboni et al., 2020; Guo et al., 2019]. The hidden states involve information from previous steps and the new inputs of all variables, but none of these methods explicitly model the immediate impact of fresh information and the long-term effects of historical data. In addition, whether the attention mechanism can be directly applied to model interpretation is still controversial [Sun et al., 2021; Serrano and Smith, 2019]. Moreover, most of the explainable studies just make predictions once based on historical data and ignore the importance of real-time series forecasting, which predicts at each time point. For example, predicting electricity consumption at each hour is crucial for electric power providers to maximize resource utilization and cut costs.

To address these challenges, we propose DeLELSTM: Decomposed-based Linear Explainable LSTM, which simultaneously: (i) identifies the instantaneous influence and longterm effects of each variable; (ii) leverages the linearity of linear regression to make interpretation transparent and straightforward. Specifically, DeLELSTM involves two components. One is standard LSTM, in which the hidden state Ht encapsulates information of all variables until time t. The other is tensorized LSTM [He et al., 2017], where the hidden state ht is a matrice and each row of the hidden state only encodes information exclusively from a certain variable of the input. To capture the instantaneous influence and long-term effects of each variable, the linear combination of the hidden state at time t-1 and the dynamic change of the hidden state at time t of each variable is used to approximate the Ht. The linearity of the linear regression can make the explanation transparent and clear. In addition, given the significance of real-time forecasting in time series, we aim to build an explainable model for real-time series forecasting.

The major contributions of this work are as follows:

• We consider the instantaneous influence and long-term effects of each variable, which commonly exists in time series data.
• We take advantage of the explanation of the linear regression model to make a clear and transparent explanation.
• Extensive experiments with three benchmark datasets show that DeLELSTM can achieve competitive performance and provide transparent explanations relative to domain knowledge.

The rest of the paper is organized as follows. Section 2 reviews the related work. Section 3 introduces the problem definition and details our proposed framework. Then we evaluate our method by comparing it with several baselines in Section 4 and conclude in Section 5.

2 RelatedWork

In recent years, different classes of approaches have been proposed to explain time series forecasting models, especially RNNs. One general approach is post-hoc analysis, which explains the models by evaluating the importance of each variable in the predictions. The other is ante-hoc methods, which incorporate interpretability directly into their structures.

Post-hoc methods frequently adopt variable-level attribution interpretations, also known as salience maps [Ding et al., 2022]. These methods assign a relevance score to each variable, indicating how sensitive a variable is to the output. Gradient-based and perturbation-based methods are the two main types of attribution methods. Gradient-based methods examine the characteristics to which output was most sensitive [Ancona et al., 2017; Shrikumar et al., 2017; Sundararajan et al., 2017; Yang et al., 2018]. In perturbation-based approaches, the variable importance is obtained by perturbing the variables with mean value or random uniform noise, running a forward pass on the new put, and comparing the difference to the original output [Dabkowski and Gal, 2017; Fong and Vedaldi, 2017]. However, these post-hoc explainable models have been criticized for failing to capture the sequential dependencies and clarifying how the underlying model arrived at a specific prediction [Tonekaboni et al., 2020; Rigotti et al., 2021].

On the other hand, ante-hoc models can provide intrinsic explanations by building self-explanatory systems. Among ante-hoc models, attention-based interpretable models are widely used for explaining RNNs. The parameters of these models, known as attention weights, are utilized to explain how the models behave over time. For example, [Choi et al., 2016] proposed RETAIN, an explainable model based on a two-level attention mechanism. Two sets of attention scores are used to identify relevant clinical variables and influential hospital visits, respectively. [Guo et al., 2019] proposed a mixture attention framework to get variable importance and temporal importance. However, the focus of the current attention mechanism is on hidden states, which encode information from both past and new observations. Differentiating between the immediate impact of new information and the long-term effects of historical data can be difficult. In addition, attention's interpretability is still debatable [Sun et al., 2021], and other studies found that attention patterns cannot reliably provide transparent and reliable explanations [Jain and Wallace, 2019].

In this paper, we propose a decomposition-based method to decompose hidden states at each time step into two components, representing instantaneous information and historical information, respectively. In addition, the linear regression model is utilized to build a transparent and faithful explainable model for time series forecasting.

3 The Proposed Framework - DeLELSTM

3.1 ProblemDefinition

Let X ∈ RD×T be a sample of a multivariate time-series data where D is the number of variables with T observations over time. Further, xt ∈ RD is the set of all observations at time t ∈ [1, · · · , T], denoted by the vector [x1t , x2t , · · · , xDt ] and X1:t ∈ RD×t is the matrix [x1; x2; · · · ; xt], representing the observations until t.

Let Y ∈ RT be the target time series of length T. Noted, the target series can be one of the multivariate time series X or not. Given X1:t ∈ RD×t , we aim to learn a function F

to predict the next value of the target series, namely, yt+1 = F(X1:t)

3.2 Network Architecture: DeLELSTM

This subsection first describes the framework of our proposed model DeLELSTM, then follows with the details of the model and the process of obtaining the interpretation.

DeLELSTM consists of two components (i) a standard LSTM, in which xt is the input at time t andHt is the hidden state at t. The hidden state Ht incorporates new information and past information of all variables; (ii) a tensorized LSTM, in which the hidden state ht is a matrix and each row of the hidden state ht only contains information taken from one particular input variable. ht-1 stands for the information from the past, and ht - ht-1 is the dynamic change brought by the new observations at time t. Here, we define ∆ht = ht - ht-1. To capture both the immediate influence and the long-term effect of each variable, we propose to approximate Ht as a linear combination of ht-1 and the dynamic change ∆ht, thereby allowing the separation of output states into contributions from instantaneous influence and long-term effects of each variable. Finally, the approximated hidden state ˆHt is used to predict the yt+1 and the computation of the hidden state in the next step Ht+1. Figure 1 shows the architecture of the proposed DELeLSTM framework.

The architecture of DeLELSTM is made to learn a representation of the multivariate time series data that suffices for accurate real-time prediction and offers a transparent explanation of each variable's long-term effect and instantaneous influence. We now proceed to illustrate each module of DeLELSTM in more detail.

Standard LSTM

The standard LSTM is shown in Equation (1).

where xt ∈ RD is the new observations of D variables at time t, Ht ∈ RM is the hidden state at time t which encodes information from all input variables until time t. M is the dimension of the hidden state. ⊙ is the elementwise product.

Tensorized LSTM

The tensorized LSTM is used to explain and approximate the hidden state Ht and identify the contribution of each variable. Here, the tensorized LSTM can be considered as a set of parallel LSTMs, each of which processes a single variable series. The hidden state of the tensorized LSTM is a matrix, and each row of the hidden state incorporates information only from a particular variable. Equation (2) shows the tensorized LSTM.

where ht = [h1t , · · · , hDt ]T , ht ∈ RD×M ,hdt ∈ RM . The hdt is the hidden state vector specific to the d-th input variable at time t, and the ht is the hidden state of all variables at time t. Uc = [u1c , · · · , uDc ], Uc ∈ RD×M×M , udc ∈ RM×M , is the hidden-to-hidden transition. Wc = [w1 c , · · · ,wD c ], Wc ∈ RD×M×1 , wd c ∈ RM×1 , is the input-to-hidden transition. (Ui, Uf , Uo), ( Wi,Wf ,Wo) have the same shapes as Uc,Wc, respectively. ⊛ is the tensor-dot operation, which is defined as the product of two tensors along the D axis, e.g., Uc ⊛ ht-1 = [u1c h1t-1 , · · · , uDc hDt-1 ]T , udc hdt-1 ∈ RM . Such a design can guarantee that gates and memory cells are also matrices, and each row of these matrices only captures the information from a single variable.

Decomposition

Ht involves the information from previous steps and new input of all variables, so it is challenging to measure the influence of each variable, including the long-term effect from Ht-1 and the instantaneous impact from xt.

Theoretically, each row of ht-1 corresponds to the information belonging to a particular variable until time t-1. Each row of ∆ht represents a single variable's new information of time t, We propose to decompose Ht as a linear combination of ht-1 and ∆ht, so that we can obtain each variable's instantaneous influence and long-term impact.

Specifically, we approximate Ht using Equation (3):

This approximation is minimised following a least squares criterion, which has a well-known solution, ˆαt, ˆβt, and corresponding optimal approximation ˆHt. Here, we use ˆHt to get the prediction of target yt+1 and compute the hidden state Ht+1.

3.3 Learning to Interpret

To get the immediate impact and the long-term effect of each variable, we derive significance from the magnitude of linear approximation weights rather than only focusing on the largest values. We first take the absolute value of ˆαt, ˆβt and get ˜αt, ˜βt after normalizing weights at each time step. Then, we propose several measures to interpret the prediction model for real-time series forecasting.

Definition 1. The instantaneous importance of the d-th variable at time t, Indt , is defined as the Equation (4), the ratio

Algorithm 1 The Training Process of DeLELSTM

Input:Time series X1:T , where T is the max time length; Set: Ht = H0, ht = h0 Output: α1:T , β1:T , y2:T+1

• 1: for t = 1 to T do
• 2: Ht=standard LSTM(xt, Ht-1);
• 3: ht=Tensorized LSTM(xt, ht-1);
• 4: ∆ht = ht -ht-1; % Compute the dynamic change of hidden state ht from t - 1 to t
• 5: (αt, βt)=Least Square(Ht, ht-1,∆ht); % Compute the optimal solution of (αt, βt) using Leaset Square
• 6: ˆHt = (ht-1)T αt + (∆ht)T βt % Compute the approximated hidden state ˆHt
• 7: ˆ yt+1=MLP( ˆHt)
• 8: end for
• 9: return α1:T , β1:T , y2:T+1

from 2010.1.1 to 2014.12.31. PM2.5 is the target series. Aside from PM2.5 values, the meteorological variables include dew point, temperature, pressure, wind direction, wind speed, hours of snow, and hours of rain. Given the measurements, the task is to forecast PM2.5 each hour within a day. For example, using the data before 2:00 predicts PM2.5 at 2:00; using the data before 3:00 predict its value at 3:00, etc.

Electricity [Gao et al., 2022]: It records the time series of electricity consumption in the US, from 2017.10.11 to 2020.6.24. The consumption is chosen as the target series sampled hourly. The other 15 time series are exogenous factors, including max temperature, min temperature, visibility, etc. Similarly to PM2.5, our task is also to forecast consumption each hour of the day.

Exchange [Lai et al., 2018]: It is the collection of the daily exchange rates of eight foreign countries, including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore, ranging from 1990 to 2016. We consider the

of ˜ βd t and (˜αd t + ˜ βd t ). Accordingly, the long-term effect of the d-th variable at time t is 1 - Indt .

Indt can help us figure out which is more important: past information until time t - 1 versus current data at time t of the d-th variable. If Indt is close to 1, we can know that the past information of the variable d has little impact on the prediction; in other words, it has no long-term effect.

Definition 2. The global importance of the d-th variable at time T, GldT , is defined as the Equation (5), which considers both the immediate impact and long-term impact of the d-th variable.

The global importance of each variable can help us identify the important variables in general. Further, we can also get the dynamic shift in a variable's weight by considering √( ˜αd t 2 + ˜ βd t 2 ) over time, which we use to provide further model interpretability.

Algorithm 1 summarizes the proposed procedure.

4 Experiment

In this section, we describe our experiments to evaluate the prediction performance and the interpretation ability of our proposed model 1 .

4.1 Datasets

We used three publicly available real-world multivariate time series datasets covering meteorology, energy, and finance fields.

PM2.5 [Liang et al., 2015]: It contains hourly PM2.5 data and the associated meteorological measurements in Beijing time series of 30 days as a sample for this task. The Singapore exchange is taken as the target series, and we aim to predict the exchange rate of Singapore each day of a month.

1 Code and supplementary is available in the repository: https://github.com/wangcq01/DeLELSTM.

4.2 Baseline Methods and Evaluation Setup

We evaluate the prediction performance of DeLELSTM with several baseline models as follows:

LSTM:[Hochreiter and Schmidhuber, 1997]: LSTM network with one hidden layer is used to learn an encoding from multivariate time series data and make predictions at each time step. The prediction performance is also used as the basis for other models.

RETAIN:[Choi et al., 2016]: RETAIN is a two-level neural attention model for sequential data that can recognize the influential events and relevant characteristics within these events.

IMV-LSTM:[Guo et al., 2019]:IMV-LSTM explores the structure of LSTM network to learn variable-wise hidden states and separate the contribution of variables to the prediction. With hidden states, a mixture attention mechanism is explored to model the generative process of the target. It has two realizations of IMV-LSTM, i.e., IMV-Full and IMVTensor. We consider both versions.

We implemented the proposed model and deep learning baseline methods with Pytorch. We used Adam[Kingma and Ba, 2014] optimizer. We conduct the grid search to select optimal parameters. The batch size is selected in {32, 64, 128}. Learning rate is searched in {0.05, 0.01, 0.001}. The size of the hidden states is selected in {32, 64, 128}. We train the models using 75% of the samples, and 15% of the samples are for validation. The remaining 10% is used as the test set. We repeat the experiment five times and report the average performance with standard deviation.

We consider three metrics to measure the prediction performance, i.e., RMSE, MAE, and MAPE. RMSE is defined as RMSE = √∑n (yn - ˆ yn)2 /N. MAE is defined as MAE = ∑n ∥yn - ˆ yn∥/N. MAPE is defined as MAPE = ∑n (|ˆ yn - yn|/|yn|)/N × 100%, where ˆ yn is the predicted value, and yn is the true value.

4.3 Prediction Performance

We compared the proposed DeLELSTM with four baseline models on real-time series forecasting and reported the results in Table 1. As shown in Table 1, among the attentionbased models, the performance of RETAIN is better than IMV-LSTM in most cases. The IMV-Full is better than IMVTensor. Our proposed model presents comparable performance and obtains the best performance in electricity consumption prediction, indicating that our decomposition-based linear explainable model can still guarantee the prediction performance, while capturing the instantaneous and longterm effects and providing transparent and clear interpretation.

4.4 Interpretation

In this subsection, we depict three case studies designed to evaluate the effectiveness of DeLELSTM in providing insightful explanations of its forecasting. In particular, we qualitatively analyze the immediate and long-term impact of each variable identified by the defined measurement lndt and 1-lndt . We also report the meaningful variables at each time step and overall according to the defined measurement GldT .

Case Study I: Electricity Data

Predicting real-time electricity consumption accurately is quite important for electric power providers, so that they can more precisely manage resources for energy generation to maximize resource utilization, cut costs, and advance the development of smart grids. Figure 2 shows how the longterm and instantaneous (short-term) impacts change throughout the day. Here, we depict the change of the top 10 features, ordered by their overall importance for predicting electricity consumption. We can observe that compared with the shortterm effects, the long-term information of most features is vital when people are sleeping and not engaged in other activities, and gradually diminishes importance during the daytime. On the other hand, the long-term effects of WindChill are important for electricity prediction.

Figure 3 depicts the dynamic change of variable importance for the electricity consumption forecast. It is seen that the electricity consumption itself has an evident autocorrelation and contributes more to the prediction. In addition, as evening approaches, pressure and precipitation become more critical. While the temperature becomes important for electricity consumption at noon, because the temperature at noon tends to be high, people need to switch on the air-conditioner to have a comfortable environment. It is a primary reason that affects electricity consumption at noon. In general, variables consumption, precipitation, temperature, pressure, wind speed and humidity are highly ranked by DeLELSTM.

Case Study II: PM2.5 Data

PM2.5 is fine inhalable particles, with diameters that are generally 2.5 micrometers and smaller. Exposure to such fine particles has been linked to early death from heart and lung disease [Franklin et al., 2008]. Understanding influential variables and forecasting PM2.5 accurately are necessary so that people can avoid going outside on time with high PM2.5 levels. We report the long-term and short-term effects of each variable during the day in Figure 4.

As shown in Figure 4, the impact of prior PM2.5 information is more significant than that of recent fresh information for real-time forecasting PM2.5 in a day. It indicates that PM2.5 always has long-term effects; we should utilize both past and current data of PM2.5. On the other hand, for other variables, such as dew point (DEWP), hours of snow(IS), temperature (TEMP), and the hours of rain (IR), the past information is only helpful before 6:00 am. After that, the ratio of instantaneous influence occupies almost 99% for predicting PM2.5 in the next hour, showing little long-term effect. Therefore, we can conclude that it is sufficient to see the current data of these variables for forecasting PM2.5 during the daytime. One possible reason is that the natural environment has a significant impact on PM2.5 changes when there is less human activity and less pollution released at night, but human activity plays an important role during the day, so the long-term effects of these meteorological variables are lost.

Figure 5 shows the dynamic change of variable importance considering the long-term effect and instantaneous influence. As shown in Figure 5, in the early morning, the PM2.5 itself contributes more to predict PM2.5 in the next hour. In the daytime, the wind speed, dew point, temperature, all take on greater significance. Taking into account all time steps, the top four important variables are the wind speed, PM2.5, the dew point, and the temperature. According to [Pu et al.,

2011], wind speed has a significant impact on the amount of such inhalable particles transported and dispersion between Beijing and its surrounding areas. In addition, [Liang et al., 2015] also conclude that dew point and temperature are critical factors for PM2.5 prediction. Therefore, our variable importance is in line with the domain knowledge.

Case Study III: Exchange Data

Investors need to be very aware of changes in foreign exchange rates. These changes greatly impact the returns on foreign investments. As a result, if the exchange rate can be predicted with accuracy, investors could improve the timing of their foreign investments and earn higher returns. The long-term and short-term effects evolving over time for predicting Singapore's exchange rate on the next day is shown in Figure 6. The exchange rates of Japan and Switzerland have both an immediate and long-term effects. Nevertheless, for other countries, the instantaneous influence is gradually increasing compared with the long-term effect.

Figure 7 displays the variable importance changing over time. Singapore has the biggest influence on predicting Singapore's exchange rate on the next day, followed by Australia, China, and Japan.

Figure 5: PM2.5 variable importance.

In summary, three case studies demonstrate that our proposed DeLELSTM can not only provide a transparent and clear explanation but also be able to distinguish the instantaneous influence and long-term effect of each variable. These explanations aid us in better understanding the data and building models for time series prediction.

4.5 Baseline Comparisons

In this subsection, we compare the interpretation results of the three baseline models, i.e., RETAIN, IMV-Full, and IMVTensor. Due to the page limitation, we show the variable importance at each time of three models in the supplementary.

As shown in Figure 1,2,3 in the supplementary 1 , we can observe that the interpretable models based on the attention mechanism give the variable importance always the same at most of the time points. In addition, except for the most important variables identified by the models, it is hard for them to distinguish the importance of the other variables. The reason might be the hidden state representations are similar across time steps [Mohankumar et al., 2020].

Table 2: Performance based on top 50% important variables

To further evaluate the sufficiency of our explanation model, we retain the top 50% of features identified by each explainable model, and then feed them into the LSTM of the same architecture to obtain the prediction results. Table 2 shows the prediction results on the electricity consumption task. We can observe that features selected by our model can obtain better performance.

Compared with these baseline models, our proposed model is able to identify the dynamic impacts of variables on the prediction over time, and distinguish the instantaneous influence and the long-term effects of each variable.

5 Conclusion

Explaining the time series forecasting model is of significance, especially in high-stakes applications. In this work, we propose DeLELSTM, a decomposition-based linear explainable LSTM, to improve the interpretability of LSTM. Specifically, DeLELSTM decomposes the hidden states into the linear combination of the past information and the new information of each variable so that it can capture the instantaneous influence and long-term effects. The utilization of linear regression also guarantees that the explanations are transparent and clear. The experimental results on three real datasets demonstrate the effectiveness of DeLELSTM compared with baseline models. The case studies show that the explanations made by DeLELSTM are in line with domain knowledge.

One limitation of this work is that the contributions to the prediction are for individual variables, ignoring the complex interactions between variables, which is common in real life. We keep this challenging task as our future work.

Ethical Statement

There are no ethical issues.

Acknowledgments

The work described in this paper was partially supported by the InnoHK initiative, The Government of the HKSAR, and the Laboratory for AI-Powered Financial Technologies. Qi WU acknowledges the support from The CityU-JD Digits Joint Laboratory in Financial Technology and Engineering; The Hong Kong Research Grants Council [General Research Fund 14206117, 11219420, and 11200219]; The CityU SRGFd fund 7005300, and The HK Institute of Data Science.