DyAt Nets: Dynamic Attention Networks for State Forecasting in Cyber-Physical Systems

1 Introduction

Cyber physical systems like the electric grid and industrial and chemical plants often have a complex set of interdependent process that are operating simultaneously. In order to ensure uninterrupted and smooth function of the CPS system, it is important for systemmaintainers to quantify the system state well into the future so as to gain enough lead time to plan for contingencies. For example, electric utilities perform weekly forecasts to estimate the expected power demand over the coming week. Such a week long forecast affords the utility enough time to increase (or decrease) the power production to meet the expected future power demand, ensuring optimal grid operation [Carvallo et al., 2017].

1 https://github.com/nmuralid1/DynamicAttentionNetworks

CPS state forecasting for chemical and industrial plants allows system maintainers to estimate the operational load the plant will undergo over the course of the forecast and hence serve as an important tool in system efficiency and health monitoring. Critical infrastructure CPS also run the risk of cyber attacks from malicious entities whose goal is to disrupt optimal system operation. CPS state forecasting forms the core of early warning systems which alert system maintainers if the system has veered from its expected state of normal operation [Filonov et al., 2016; Malhotra et al., 2016]. Traditional forecasting models predict system state at the next time step given system measurements at the current time step. However, as mentioned previously, CPS state forecasting contexts require that the forecasting models yield predictions multiple steps into the future; CPS traditionally have components which share highly non-linear relationships evolving continuously and the underlying process might not be stationary or easy to define mechanistically. Hence, traditional auto-regressive models or dynamical systems like Kalman filters might be ineffective in capturing the data representation. However, deep learning models have proven effective in learning highly non-linear function spaces and require no stationarity assumptions to be satisfied or mechanistic representations of system operation to be provided. A popular deep learning architecture for sequence prediction (employed extensively for neural machine translation and abstractive text summarization) is the Sequence to Sequence (Seq2Seq) architecture a.k.a encoder-decoder architecture.

Seq2Seq models can also be adopted for time series sequence forecasting applications. Most existing work on deep learning for time series [Filonov et al., 2016; Malhotra et al., 2016] makes use of a traditional Seq2Seq architecture for forecasting future values of the time series without any modifications. Such traditional Seq2Seq architectures usually also employ recurrent units for each encoder and decoder step and an attention mechanism wherein the hidden state for each decoder unit is constructed as some combination of a set of hidden states from preceding units. Thus far, Seq2Seq models used for time series forecasting have employed standard attention mechanisms where each decoder unit only considers the hidden states from the encoder phase. Such an attention mechanism is effective for neural machine translation or abstractive text summarization because, in these applications, all the information required for sequence generation, is encapsulated in the encoding phase. However, such an encoderfocused attention mechanism is not optimal for time series forecasting where each decoder unit models one time step of a sequence in an ever-evolving temporal process, all of whose information is not captured by the encoder phase. Time series models are known to benefit significantly from knowledge of the recent past. Thus, traditional encoder-focused attention mechanisms in Seq2Seq forecasting models lose valuable information as states from previous decoders are not considered by an attention mechanism that only considers encoder hidden states. In this paper, we introduce two variants of novel hierarchical dynamic attention models for multivariate time series forecasting which addresses this issue. Our contributions are as follows:

• We introduce a novel dynamic attention (DyAt) mechanism for effective multivariate time series forecasting with Seq2Seq models.
• We develop two novel hierarchical dynamic attention models, Hierarchical Dynamic Attention Networks (DyAt-H) and Hierarchical Dynamic Attention Networks with Max Pooling (DyAt-MaxPool-H).
• We also characterize the performance of our hierarchical DyAt models relative to other state of the art time series forecasting models and discuss the properties responsible for our models yielding superior performance.

In section 2, we formally introduce the multivariate time series forecasting task in the context of Seq2Seq models and develop our DyAt models. The experimental setup is described in section 3, followed by discussion of experimental results in section 4. We then present a brief survey of related literature in section 5 followed by concluding remarks in section 6.

2 ProblemFormulation

Let us consider a set D = {X1,X2, ..,Xm} where each Xi ∈ Rl×k represents a multivariate sequence of length l with k time series at each step of the sequence. Sequence forecasting applications in time series employ encoder-decoder (Seq2Seq) models wherein the encoder is supplied with a sequence Xi and the learning task is for the decoder to forecast the next k-variate sequence Xi+1 of length l.

2.1 Seq2Seq AttentionModels

Formally, in traditional Seq2Seq architectures with attention, the prediction at each decoder step is calculated as defined by [Luong et al., 2015].

Let E = {he1 , he2 , .., hen } represent hidden states of the n encoder units and D = {hd1 , hd2 , .., hdq } represent hidden states of the q decoder units of a Seq2Seq model, where each h∗i ∈ Rh×1 comes from a recurrent unit like LSTM, GRU, basic RNN etc. The prediction at each decoder is calculated as defined in Eq. 1.hdi = f(hdi-1, ˆ

In Eq. 1, f may be a recurrent unit like an RNN, LSTM or a GRU. hdi-1 represents the hidden state from the previous decoder unit and ˆ yi-1 ∈ Rk×1 the k-variate prediction of the previous decoder unit passed as input to the current decoder unit. αi∗ ∈ Rl×1 is the attention weight vector for decoder i over the encoder hidden states (i.e l = |E|) and ci ∈ Rh×1 is the attentional context vector calculated as a linear combination over all E, and corresponding attention weights αi∗.

Attention Calculation. The attention energy vector αi∗ for the ith decoder unit can be calculated in multiple ways. The attention energies can be calculated purely based on the current hidden vector or as a function of the alignment between the current hidden vector and encoder states. In Eq. 2, we define two such variants of the scoring mechanism for calculating attention energies.

The attention energies are then normalized using a softmax transformation, to obtain the attention weights αi∗ ∈ Rl×1 used to obtain the attentional context vector ci.

Next, the attentional hidden state ˜hdi of the ith decoder is obtained by imposing a non-linear transfer function (hyperbolic tangent) over an affine transformation (with Wh ∈ Rh×2h ) of the concatenation of ci and hdi . Finally, g is some function that calculates the ith prediction ˆ yi from the corresponding decoder attentional hidden state ˜hdi . In our experiments we consider g to be a simple linear layer.

2.2 Dynamic Attention Networks

In time series applications, we hypothesize that calculating attention just over the encoder hidden states is insufficient. This is because, unlike in the case of NMT where all the information required by the decoder sequence is contained in the encoder steps, the time series forecasting task models an evolving process and each step in the decoder phase also carries with it important information about function evolution. Hence, incorporating these signals into the attention mechanism is paramount to learning better time series forecasting models. To improve the learned representation of Seq2Seq architectures in the context of multivariate time series forecasting, we introduce the Dynamic Attention network (DyAt). A DyAt employs a Seq2Seq architecture with a novel dynamic attention mechanism.

Let us consider a set of hidden states Hi = {h(i-1):(i-l)} where each hidden state can either be an encoder or decoder hidden state from a previous time step, i.e., (hk ∈ E ∨ hk ∈ D) ∀hk ∈ Hi. The calculation of the context vector ci and the corresponding attentional hidden state ˜hdi is modified as defined in Eq. 3.

Dynamic Attentional Context Vector. In Eq. 3, each candidate hidden state hj in the linear combination to produce the dynamic attentional context vector ˜ ci, no longer originates from the set of encoder hidden states E as is the case for the static attentional context vector ci in Eq. 1. Rather, ˜ ci is derived from set Hi which consists of the past l hidden states irrespective of whether they are from the encoder or the decoder. Maintaining a dynamic candidate set of hidden states Hi for each decoder unit i, allows the model to more effectively attend to the model properties learned in the recent past as opposed to only being restricted to focus on the model properties learned during the encoder phase.

2.3 Hierarchical Dynamic Attention Networks

In the aforementioned DyAt model, as we move deeper into the decoder forecasting sequence, we can observe that the set Hi becomes dominated by a majority of hidden states from previous decoders and hence information from the encoder phase (which receives ground-truth input) considered in the decoder phase is continually reduced. Due to the lack of ground truth inputs in the decoder phase, the effect of erroneous predictions or bad hidden representations increases as we traverse deeper into the decoder sequence.

To alleviate this effect and encourage the DyAt models to learn a more holistic representation, we introduce Hierarchical Dynamic Attention Networks (DyAt-H) which also considers a concise representation of the encoder referred to as the encoder context vector ce in addition to the dynamic attentional context vector ˜ ci at each decoder unit. A schematic representation of the DyAt-H model is presented in Fig. 2.

In DyAt-H models, at the outset of the decoder phase, we generate ce ∈ Rh×1 using the contextGen(·) function as defined in Eq. 4. Context vector ce is calculated using all encoder hidden states E and is calculated only once per decoder phase. We experimented with two variants for generating ce, detailed in Eq. 5.

The attention energies (αi∗), attention weights (αi∗), dynamic attentional context vector (˜ ci) and dynamic attentional hidden state (˜hdi ) for the ith decoder unit are all calculated as detailed in section 2.1 and section 2.2. Finally, the blend function is used to produce the hierarchical attentional hidden state ˜hhier i by combining the encoder context vector ce with the dynamic attentional hidden state ˜hdi which is then supplied to the g(·) function to obtain the k-variate prediction at time step i. The blend function can take multiple forms, it can be an affine transform over the concatenation of the two vectors or the result of a pooling operation over the two. For our experiments, we use mean-pooling.

It can be argued that attention mechanisms (static or dynamic) are redundant as standard recurrent units are designed to propagate model properties effectively to subsequent recurrent units. However, as we showcase in our results, in longterm forecasting tasks, hierarchical dynamic attention mechanisms do indeed provide forecasting models the added ability to focus on relevant parts across the entire sequence, both in the immediate and the distant past.

3 Experimental Setup

We demonstrate the forecasting performance of DyAt and DyAt-H on several state of the art datasets given below:

Electricity 2 : This dataset contains electricity consumption in kilo-watt hour (kWh) measured in 15 minute intervals from 2011-14 for n=370 clients. For our purposes, we randomly choose 20 clients to train and test our models.

Tennessee Eastman Challenge Process (TEP) 3 , represents a CPS performing a chemical process. It is a popular benchmark dataset in process control studies and for system identification, forecasting and anomaly detection applications [Filonov et al., 2017; Juricek et al., 2001].

Gasoil Heating Loop (GHL) 4 released by [Filonov et al., 2016], is another benchmark dataset for multivariate system state forecasting and anomaly detection in CPS. It represents a chemical process of a fluid (gasoil), being heated to a particular temperature and transported over to a collection tank once the fluid is at the specified temperature.

Both GHL and TEP were created via process based simulations. Different time series within them show different periodicity, trends and inter-dependencies. For example, in GHL the receiving tank temperature and the heating tank temperature are correlated. In the Electricity dataset, given that each time series corresponds to the electricity consumption of one client, they typically have a periodicity of 24 hours. Thus, evaluating our model on these datasets corresponding to diverse and complex scenarios showcases the effectiveness and generalization capacity of our model architecture.

3.1 Methods for Comparison

Traditional Regression Models

1. GPR stands for Gaussian process regression for time series modeling [Roberts et al., 2013].
2. LSVR represents the vector-autoregression model with linear support vector regression objective function. [Vapnik et al., 1997].

Deep Learning Models

1. We also compare our models against the model proposed by [Filonov et al., 2016]. The model is a two layer LSTM Seq2Seq model with static attention.
2. LSTNet, proposed by [Lai et al., 2018], is a state-ofthe-art multivariate forecasting model consisting of recurrent, convolutional and auto-regressive components.
3. S2S represents a traditional Seq2Seq model.
4. S2S + Attn. represents a Seq2Seq model with the traditional static attention mechanism.

2 https://tinyurl.com/yyr4lo9j

3 https://tinyurl.com/y2ej46hr

4 https://tinyurl.com/yy8vtla5

Proposed Dynamic Attention Models

1. DyAt: A Seq2Seq model with a dynamic attention mechanismwithout the hierarchical component, as outlined in section 2.1 and 2.2. Attention weights are calculated using the location-based score(·) function (Eq. 2) .
2. DyAt-H: A Seq2Seq model with a hierarchical dynamic attention mechanism as outlined in Eq. 4. The mean based contextGen(·) function is employed with the location based score(·) function.
3. DyAt-MaxPool-H: Another Seq2Seq model with a hierarchical dynamic attention mechanism similar to DyAtH, except in this case, the learn based contextGen(·) function is employed to generate the encoder context vector ce and the bilinear score(·) function is employed instead of the location-based variant.

Evaluation Metrics: We evaluate our models using two metrics: (1) Mean-Squared Error (MSE) and (2) Weighted MSE, wherein the later decoder steps are weighted higher. The two metrics are formally defined as

3.2 Experimental Setting &Model Architecture

We arbitrarily choose 4 different forecasting sequence lengths 90, 110, 130 to test model performance on long-term forecasting and length 10 for short-term forecasting. We don't consider sequence length to be dataset specific, in an effort to maintain uniform experimental settings across datasets but, in practice, experts can provide optimal values of sequence lengths for forecasting tasks. We employ Seq2Seq models with a single hidden layer, GRU as the recurrent unit and set the encoder and decoder phases to have the same sequence length (l) for simplicity. However, our proposed approach is applicable even if this constraint is relaxed. All Seq2Seq models are trained with early stopping for a maximum of 200 epochs. Finally, each experiment has a 80/20 training/validation split and a separate holdout test set.

4 Results &Discussion

We evaluate the forecasting performance of our DyAt models in four experimental settings by varying the length of the prediction sequence between 10 and 130 time steps for all the models. For each setting, we evaluate the models on three different datasets. We also compare our DyAt models to several state of the art forecasting models as outlined in section 3.1.

The forecasting performance results have been detailed in Tab. 1. The best performing hierarchical DyAt model (either DyAt-H or DyAt-MaxPool-H) was able to achieve at least a 17.82% improvement over the state of the art [Filonov et al., 2016] Seq2Seq forecasting model and 21.88% improvement over the LSTNET model proposed by [Lai et al., 2018] across all the datasets and sequence lengths. The DyAtMaxPool-H model outperforms all other models in the CPS

state forecasting tasks on the GHL, TEP datasets (minimum performance improvement 13.69%). However, in the case of the electric load forecasting, DyAt-H is the best performing model (minimum performance improvement 18.83%). We believe this can be attributed to the well known property of the maxpool operation (used to calculate the encoder context vector ce in the DyAt-MaxPool-Hmodel) leading to order invariance [Scherer et al., 2010]. Due to this, the short-term cyclical changes in the electricity dataset are not captured as effectively by the context vector ce in the case of DyAtMaxPool-H leading to the drop in performance relative to DyAt-H which employs a mean operation to generate ce instead. In the case of CPS state forecasting, the transitions of state are much more gradual and the data is devoid of significant short-term cyclical components. In order to showcase the effectiveness of the DyAt mechanisms, we also compare their performance to standard Seq2Seq architectures with and without the traditional attention mechanisms. Our DyAt-H and DyAt-MaxPool-H models achieve a minimum improvement of 31.33% over the Seq2Seq(S2S) model without attention and a minimum performance improvement of 13.69% over traditional Seq2Seq attention (S2S + Attn.) models.

4.1 Sequence Forecasting Performance Inspection

To further showcase the sequence forecasting performance of DyAt models, we also calculate the MSE at each decoder time step and plot the average MSE per decoder unit for each Seq2Seq model in Table 1. We showcase results on the GHL dataset and notice that DyAt-H and DyAt-MaxPool-H, comfortably outperform the other models as depicted in Fig. 3. We notice that although in the short-term forecasting tasks (sequence length = 10) there doesn't seem to be too much disparity in forecasting performance over the entire sequence length, the disparity in performance increases in the case of longer sequences and the performance gap between DyAt-H, DyAt-MaxPool-H models and others increases as we move deeper into the prediction sequence. This showcases the ability of the proposed hierarchical models to reduce the effect of error-propagation to subsequent steps. The hierarchical nature of DyAt-H and DyAt-MaxPool-H models helps them consider the rich information from the encoder phase in addition to the dynamic attention mechanism which allows them to consider representations from the immediate past. Hence the hierarchical attentional hidden state ˜hhier i at each decoder captures this rich representation from both the encoder phase as well as the recent past leading to better long-term forecasts.

4.2 Evolution of Hierarchical Attentional Vector

As outlined in section 2.3, the hierarchical attentional hidden state ˜hhier i is calculated as a function of the encoder context vector ce and the dynamic attentional hidden state ˜hdi at each decoder. In order to further understand the effect of the hierarchical model, we inspected the evolution of ˜hhier i over the course of a prediction sequence with respect to the encoder context vector ce and the dynamic attentional hidden state at each decoder ˜hdi . We calculate the cosine similarity of ˜hhier i and ce, let's call this Simce and similarly Sim˜hdi for each prediction sequence. We then average the cosine similarity over all batches for each decoder step and plot the Sim˜hdi vs. Simce values in Fig. 4. We only show results for sequence length 90 and 130 due to space limitations. We notice a consistent pattern for all long-term forecasting tasks (90, 110, 130) wherein as the forecast moves further into the decoder phase, the ˜hhier i starts to rely more heavily on the encoder context vector ce in addition to the dynamic attentional hidden state ˜hdi . Throughout the decoding sequence, we notice that ˜hhier i is influenced significantly by both ˜hdi and ce. This ability of the hierarchical DyAt models to utilize both encoder and dynamic attentional representations leads to their superior performance.

5 RelatedWork

Time Series Forecasting Methods: Time series forecasting is a well researched topic, with a variety of models like autoregressive models (AR, MA, ARIMA) and state-space models like Kalman and particle filters, detailed by [De Gooijer and Hyndman, 2006]. Such traditional models however, require certain domain rules governing process state transition (as in the case of Kalman filters) to be known or others like the auto-regressive models require certain assumptions made regarding the properties of the time series themselves to hold. Deep Learning for Time series Forecasting: Traditional models fail in highly non-linear time series spaces. Recently, deep learning models have proven effective in learning complex function representations. Hence, we employ deep neural network models for the task of time series forecasting. Recently, [Lai et al., 2018] proposed a deep learning model (LSTNET) employing convolutional, recurrent and auto-regressive methods for multivariate time series forecasting in the context of solar and electric load forecasting. They showcased the performance of LSTNET for both short and long-term forecasting tasks. [Romeu et al., 2013] used stacked de-noising autoencoders for indoor temperature forecasting. [Qiu et al., 2014] propose an ensemble of deep belief networks used in conjunction with a support vector regressor (SVR) for time series forecasting and other regression tasks.

Deep Learning for Sequence Modeling: Seq2Seq models are popular in the natural language processing domain. They were initially proposed for the sequential data modeling task of neural machine translation (NMT) [Cho et al., 2014b; Sutskever et al., 2014; Sennrich et al., 2016; Vaswani et al., 2017]. Variants of Seq2Seq models have been used for NMT and text summarization [Cho et al., 2014a; Lin et al., 2018; Nallapati et al., 2016; Gehring et al., 2017]. To the best of our knowledge, all Seq2Seq models used for time series forecasting in CPS, employ traditional attention mechanisms. We are the first to introduce dynamic attention for multivariate time series forecasting, through our novel DyAt models.

6 Conclusion

In this paper, we developed a novel dynamic attention mechanism and two novel hierarchical dynamic attention models for long-term multivariate time series forecasting and evaluated them on many CPS state forecasting and load forecasting datasets. We demonstrated that our hierarchical dynamic attention models achieved significant improvement in forecasting performance over state of the art time series forecasting baselines. Moving forward, we wish to incorporate more sophisticated attention mechanisms into the DyAt-H and DyAt-MaxPool-H models and inspect their performance in spatiotemporal forecasting applications.

Acknowledgments

This research was supported in part by the National Science Foundation via grants DGE-1545362, IIS-1633363.