GeoMAN: Multi-level Attention Networks for Geo-sensory Time Series Prediction

1 Introduction

There are massive sensors, such as meteorological sites, that have been deployed in the physical world. Each of them has a unique geospatial location, constantly generating time series readings. A group of sensors collectively monitor the environment of a spatial region, with the spatial correlation between their readings. We call such sensors' readings geosensory time series. Additionally, it is common that one sensor generates multiple kinds of geo-sensory time series as it monitors different target conditions simultaneously. For example, as shown in Figure 1(a), the loop detectors in roads report timely readings about the vehicles passing by as well as their travel speed. Figure 1(b) presents that the sensors generate three different chemical indexes about water quality every 5 minutes. Besides monitoring, there is a rising demand for geo-sensory time series prediction, e.g., traffic prediction.

However, forecasting geo-sensory time series is very challenging, affected by the two following complex factors:

1) Dynamic spatio-temporal correlations.

• Complex inter-sensor correlations. Figure 1(c) shows the spatial correlation between different sensors' time series is highly dynamic, changing over time. Moreover, geo-sensory time series varies by locations non-linearly. When modeling dynamic pairwise correlation, classical methods (e.g., probabilistic graphical models [Koller and Friedman, 2009]) have extremely heavy computational cost due to their massive parameters.
• Dynamic intra-sensor correlations. First, a geo-sensory time series usually follows a periodic pattern (e.g., S1 in Figure 1(c)), which changes over time and varies geographically [Zhang et al., 2017]. Second, sensors' readings sometimes fluctuate tremendously and suddenly change, quickly decreasing the impact of their previous values. Thus, how to select the relevant previous time intervals to make predictions remains a challenge.
• External factors. Sensors' readings are also affected by the surrounding environment such as meteorology (e.g., a strong wind), time of day (e.g., rush hours) and land use.

To tackle these aforementioned challenges, we propose a Multi-level Attention Network (GeoMAN) to predict the readings of a geo-sensor over a couple of future hours. The contributions of our study are three-fold:

• Multi-level attention mechanism. We develop a multilevel attention mechanism to model the dynamic spatiotemporal correlations. Specifically, in the first level, we propose a novel attention mechanism, consisting of local spatial attention and global spatial attention, to capture the complex spatial correlations between different sensors' time series (i.e., inter-sensor correlation). In the second level, a temporal attention is applied to model the dynamic temporal correlations (i.e., intra-sensor correlation) between different time intervals in a time series.
• External factor fusion module. We design a general fusion module to incorporate the external factors from different domains. The learned latent representations are fed into the multi-level attention networks to enhance the importance of these external factors.
• Real evaluation. We evaluate our approach based on two real-world datasets. Extensive experiments show the advantages of our method against all baselines.

2 Preliminary

2.1 Notations

Suppose there are Ng sensors, each of which generates Nl kinds of time series. Among them, we specify one time series as target series for making predictions, while other kinds of series are used as features. Given a time window of length T , we use Y = (y1 , y2 , . . . , yNg ) ∈ RNg×T to denote the readings of all target series during past T hours, where yi ∈ RT belongs to sensor i. We use Xi = (xi,1 , xi,2 , . . . , xi,Nl )⊤ = (xi1 , xi2 , . . . , xiT ) ∈ RNl×T as the local features of a sensor i, where xi,k ∈ RT is the k-th time series reported by this sensor, and xit = (xi,1 t , xi,2 t , . . . , xi,Nl t )⊤ ∈ RNl denotes the readings of all time series from sensor i at time t. Besides the local features of sensor i, other sensors also shareplenty of information that is useful to our predictions due to the geospatial correlations between different sensors. To this end, we combine the local features of each sensor into a set X i = {X1 , X2 , . . . , XNg } as the global features of sensor i.

2.2 Problem Statement

Given the previous readings of each sensor and the external factors, predict the readings of the sensor i over next τ hours, denoted as ˆyi = ( ˆ yiT+1 , ˆ yiT+2 , . . . , ˆ yiT +τ )⊤ ∈ Rτ .

3 Multi-level Attention Networks

Figure 2 presents the framework of our approach. Following the encoder-decoder architecture [Cho et al., 2014b], we employ two separate LSTMs [Lin et al., 1996], one to encode the sequence of input, i.e., historical geo-sensory time series, and another one to predict the output sequence ˆyi . More specifically, our model GeoMAN is composed of two major parts as follows: 1) Multi-level attention mechanisms. It consists of an encoder with two kinds of spatial attention mechanisms and a decoder with temporal attention. In the encoder, we develop two different attention mechanisms, i.e., local spatial attention and global spatial attention as depicted in Figure 2, which can capture complex inter-sensor correlations at each time slot by referring to the previous hidden state of encoder, previous values of sensors as well as the spatial information (i.e., sensor networks). In the decoder, we use a temporal attention to adaptively select the relevant previous time intervals for making predictions. 2) External factor fusion. This module is used to handle the effects of external factors, and its output is fed to the decoder as a part of its inputs. Here, we use ht ∈ Rm and st ∈ Rm to denote the hidden state and cell state of the encoder at time t, respectively. Likewise, dt ∈ Rn and s′ ∈ Rn represent those of the decoder.

3.1 Spatial Attention

Local Spatial Attention

We first introduce the local spatial attention mechanism. For a certain sensor, there is a complex correlation among its local time series. For instance, an air quality monitoring station reports different time series such as PM2.5 (Particular Matter), NO and SO2. In fact, the concentration of PM2.5 is usually affected by other time series, including other air pollutants and local weather conditions [Wang et al., 2005]. To address this issue, given the k-th local feature vector of the i-th sensor (i.e., xi,k ), we employ an attention mechanism to adaptively capture the dynamic correlation between the target series and each local feature with:

where [·; ·] is a concentration operation. Here, the learnable parameter are vl, bl ∈ RT , Wl ∈ RT×2m and Ul ∈ RT×T . The attention weights of local features are jointly determined by the input local features and the historical states (i.e., ht-1 and st-1) in the encoder. This score semantically represents the importance of each local contributing feature. Once we obtain the attention weights, the output vector of local spatial attention at time step t is computed with:

Global Spatial Attention

To a target series reported by a sensor, that of other sensors have direct impacts on it. However, the impacting weights are highly dynamic, changing over time. Since there might be many irrelevant series, directly using all kinds of time series as the encoder inputs to capture the correlations between different sensors results in very high computational cost and degrades the performance. Note that such impacting weights are affected by the local condition of other sensors. For example, when the wind blows strongly from the remote places, the air quality in a certain region is more affected by these places than it used to be. Inspired by this fact, we develop a new attention mechanism to capture the dynamic correlations between different sensors. Given the i-th sensor as our predictive target and another sensor l, we calculate the attention weight (i.e., impacting weight) between them as follows:

glt = v⊤g tanh (Wg [ht-1; st-1] +Ugyl +W′g Xlug + bg ) , where vg, ug, bg ∈ RT , Wg ∈ RT×2m, Ug ∈ RT×T and W′g ∈ RT×Nl are the parameters to be learned. By referring to the target series and local features of other sensors, this attention mechanism can adaptively select the relevant sensors to make predictions. Meanwhile, the historical information is spreading across the time steps by considering the previous hidden state ht-1 and cell state st-1 in the encoder.

Note that the spatial factors also contribute to the correlations between different sensors. Generally, geo-sensors are interconnected with each other through an explicit or underlying network. Here, we use a matrix P ∈ RNg×Ng to measure the geospatial similarity (such as the inverse of geospatial distance), where Pi,j denotes the similarity between sensor i and j. Different from the attention weight, the geospatial similarity can be considered as a prior knowledge. In particular, if Ng is too large, an alternative is by using the nearest or closest ones instead of all sensors. After that, we employ a softmax function to ensure all the attention weights sum to one, jointly considering the geospatial similarity as follows:

where λ is a tunable hyperparameter as a trade-off. if λ is large, the termwill force the attention weight to be as similar as the geospatial similarity. With these attention weights, the output vector of the global spatial attention is computed as:

3.2 Temporal Attention

Since the performance of encoder-decoder architecture will degrade rapidly as the encoder length increases [Cho et al., 2014a], an important extension is by adding a temporal attention mechanism, which can adaptively select the relevant hidden states of the encoder to produce output sequence, i.e., model the dynamic temporal correlation between different time intervals in the target series. Specifically, to compute the attention vector at each output time t′ over each hidden state of the encoder, we define:

where Wd ∈ Rm×m, W′d ∈ Rm×2n , and vd, bd ∈ Rm are learnable. These scores are normalized by a softmax function to create the attention mask over the encoder hidden states.

3.3 External Factor Fusion

Geo-sensory time series has a strong correlation with the spatial factors, e.g., POIs and sensor networks. Formally, these factors jointly feature the function of a region. Besides, there are many temporal factors affecting the readings of sensors, such as meteorology and time. Inspired by the previous works [Liang et al., 2017; Wang et al., 2018] focusing on the effects of external factors in spatio-temporal applications, we design a simple yet effective component to handle these factors.

As shown in Figure 2, we first incorporate the temporal factors including time features, meteorological features, and SensorID which specifies the target sensor. Since the weather condition at future time slot is unknown, we use weather forecasts to enhance our performance. Note that most of these factors are categorical which cannot be fed to neural networks directly, we transform each categorical attribute into a lowdimensional vector by feeding them into different embedding layers separately. In terms of the spatial factors, we use the POIs density of different categories as POIs features. Since the properties of sensor networks depend on the specific situation, we simply use the structural features of the networks, such as the number of neighbors and intersections. Finally, we concatenate the obtained embedded vectors together with the spatial features as the output of this module, denoted as ext′ ∈ RNe , where t′ is a future time step in the decoder.

3.4 Encoder-decoder &Model Training

In the encoder, we briefly aggregate the outputs from the local spatial attention and the global spatial attention with:

where ˜xt ∈ RNl+Ng . We feed the concatenation ˜xt as the new input to the encoder and update the hidden state at time t by using ht = fe(ht-1, ˜xt), where fe is an LSTM unit.

In the decoder, once we get the weighted summed context vector ct′ at a future time step t′ , we combine it with the output of external factor fusion module ext′ and the last output of decoder ˆ yi t′-1 to update the decoder hidden state with dt′ = fd (dt′-1, [ ˆ yi t′-1 ; ext′ ; ct′ ]), where fd is an LSTM unit used in the decoder. Then, we concatenate the context vector ct′ with the hidden state dt′ , which becomes the new hidden state from which we make final predictions as follows:

where the matrix Wm ∈ Rn×(m+n) and the vector bm ∈ Rn map the concentration [ct′ ; dt′ ] ∈ Rm+n to the size of the decoder hidden state. Finally, we use a linear transformation (i.e., vy ∈ Rn and by ∈ R) to generate the final output.

Since our approach is smooth and differentiable, it can be trained via back-propagation algorithm [Rumelhart et al., 1986]. During the training phase, we use a Adam optimizer [Kingma and Ba, 2014] to train our model by minimizing the mean squared error (MSE) between the predicted vector ˆyi and the ground truth vector yi ∈ Rτ at sensor i:

where θ are all learnable parameters in the proposed model.

4 Experiments

4.1 Settings

Datasets

We conduct our experiments over two different datasets as depicted in Table 1. Each dataset contains three sub-datasets: meteorological data, POIs data and sensor networks data.

• Water quality: The sensors throughout water distribution system in a city of southeast China provides the realtime water quality information every five minutes from a period of three years, e.g., residual chlorine (RC), turbidity and PH. We consider the concentration of RC as target series since it is widely employed as the major water quality index in environmental science [Rossman et al., 1994]. Totally, there are 14 sensors collectively monitoring 10 different indexes, which are interconnected through pipe networks. We use the metric proposed by [Liu et al., 2016a] as the similarity matrix in this dataset.
• Air quality: Scratched from a public website 1 , this dataset includes the concentration of many different pollutants (e.g., PM2.5, SO2 and NO), together with some meteorological readings (e.g., temperature and wind speed) collected by totally 35 sensors every hour in Beijing. Among them, the primary pollutant of air quality is PM2.5 in most cases, thus we employ its reading as the target series. We briefly use the inverse of geospatial distance to denote the similarity between two sensors.

In the experiment with respect to the water quality, we partition the data into non-overlapped training, validation and test data by a ratio of 4:1:1. i.e., we use the first two-year data as the training set, the first half of the last year as the validation set, and the second half of the last year as the test set. Unfortunately, we cannot obtain such big data in the second dataset. Hence, we use a ratio of 8:1:1 to overcome it.

Table 1: Detail of the datasets.

Evaluation Metrics

We use multiple criteria to evaluate our model, including the rooted mean squared error (RMSE) and the mean absolute error (MAE), both of which are widely used in regression tasks.

Hyperparameters

Following the previous works [Zheng et al., 2015; Liu et al., 2016b], we set τ = 6 to make short-term predictions. During the training phase, the batch size is 256 and the learning rate is 0.001. In external factor fusion module, we embed SensorID to R6 and the time features to R10 . Totally, there are 4 hyperparameters in our model, of which the trade-off parameter λ is empirically fixed from 0.1 to 0.5. For the length of window size T, we set T ∈ {6, 12, 24, 36, 48}. For simplicity, we use the same hidden dimensionality at the encoder and the decoder, and conduct a grid search over {32, 64, 128, 256}. Moreover, we use stacked LSTMs (the number of layers is denoted as q) as the unit of encoder and decoder to enhance our performance. The setting in which q = 2, m = n = 64 and λ = 0.2 outperforms the others in the validation set.

4.2 Baselines

We compare our model with nine baselines as follows:

• ARIMA [Box and Pierce, 1970]: It is a well-known model for forecasting future values in a time series.
• VAR [Ziv, 2006]: Vector Auto-Regressive can capture the pairwise relationships among all sensors with very high computational costs due to massive parameters.
• GBRT [Friedman, 2001]: Gradient Boosting Regression Tree (GBRT) is an ensemble method for the regression tasks and widely used in practice.
• FFA [Zheng et al., 2015]: A multi-view based hybrid model considers spatio-temporal dependencies and sudden change simultaneously to forecast sensor's reading.
• stMTMVL [Liu et al., 2016b]: It fuses the heterogeneous data frommultiple domains, jointly capturing the local and global information of sensors to make predictions based on multi-task multi-view learning.
• stDNN [Zhang et al., 2016]: A deep neural network (DNN)-based prediction model for spatio-temporal data.
• LSTM: We use 6 different LSTMs to forecast the readings of a sensor over the next 6 hours separately.
• Seq2seq [Sutskever et al., 2014]: It uses a RNN to encode the input sequences into a feature representation and another RNN to make predictions iteratively.
• DA-RNN [Qin et al., 2017]: A dual-staged attention model for time series prediction, which shows the stateof-the-art performance in time series prediction.

Our model, as well as the baselines, are implemented with TensorFlow [Abadi et al., 2016] on the server with one Tesla K40m and Intel Xeon E5. We consider the previous 6-hour data as the input of ARIMA. In stMTMVL and FFA, we use the default settings by their authors. Similar to GeoMAN, we use the former T = {6, 12, 24, 36, 48}-hour data as the input of other baselines. Finally, we test different hyperparameters for them all, finding the best setting for each.

4.3 Model Comparison

In this section, we compare our model with the baselines on the two datasets. To be fair, we present the best performance of each method under different parameter settings in Table 2.

In terms of water quality prediction, our proposed method clearly outperforms all the baselines on both metrics. Specifically, GeoMAN shows 14.2% and 13.5% improvements beyond the state-of-the-art approach (DA-RNN) on MAE and RMSE respectively. On the other hand, since the concentration of RC follows a certain periodic pattern, stDNN and RNN-based methods (i.e., Seq2seq, DA-RNN and GeoMAN) achieve better performance than stMTMVL and FFA by considering much longer temporal dependency. Compared to LSTM which makes predictions at each future time step separately, GeoMAN as well as Seq2seq bring significant improvements due to the positive effects of the decoder component. Remarkably, GBRT performs better against most baselines, which reveals the advantage of the ensemble methods.

Compared to relatively stable readings of water quality, the concentration of PM2.5 sometimes fluctuates tremendously, which makes it more difficult to forecast. Table 2 presents a comprehensive comparison on air quality data in Beijing. It is easy to be seen that our model achieves the best performance on MAE and RMSE simultaneously. Following the previous work [Zheng et al., 2015] that focuses on MAE, we mainly discuss on such metric. Our approach has relatively from 7.2% up to 63.5% lower MAE than these baselines, demonstrating that it has better generalization performance on other applications. Another interesting observation is that stMTMVL works well in water quality prediction but shows inferiority here since the number of joint-learning tasks of air quality prediction is much larger than that of water quality.

4.4 Variant Comparison

To further investigate the effectiveness of each model component, we compare GeoMAN with its variants as follows:

• GeoMAN-nl: There is no local spatial attention in the first-level attention (i.e., spaital attention).
• GeoMAN-ng: We simply remove the global spatial attention module from the first-level attention.
• GeoMAN-nt: To validate the temporal attention mechanism, we remove it from GeoMAN directly.
• GeoMAN-ne: This variant does not consider the effects of external factors. i.e., no external factor fusionmodule.

Evaluation on Spatial Attention

The experimental results are presented in Figure 3(a). From this figure, we observe that: 1) the combination of local and global spatial attention shows great superiority against each individual one, which demonstrates the importance of both the local and global information. 2) The fact that GeoMAN outperforms DA-RNN also verifies the advantage of our spatial attention against the input attention applied in the latter one. Despite that the local time series at a certain sensor and that from other sensors have different impacts on the target series, DA-RNN simply treats all these time series as equal and directly feeds them into the encoder to select relevant series by an input attention. It has the following two drawbacks. First, the input attention in DA-RNN cannot capture the spatial dependency between sensors. Second, the performance of DA-RNN decreases rapidly by the number of sensors.

Evaluation on External Factor Fusion

As a practical component of our model, this module provides additional information to boost the predictive performance. As illustrated in Figure 3(b), our model can significantly outperform GeoMAN-ne when predicting on more distant future over the second datset, since it allows our model to consider the temporally-related external factors in the future time step.

Evaluation on Temporal Attention

Temporal attention mechanism is employed to determine the discriminative encoder hidden state for making predictions. Hence, we attempt different encoder length T to verify its validity. As depicted in Figure 4(a), these methods follow a similar trend in water quality prediction. Most of them achieve the least errors when T = 24. Their performance will drop rapidly when T is large due to the difficulty with such long historical dependency. Different from water quality, Figure 4(b) reveals that the majority of models perform best when T = 12, because there is no such long temporal dependencies in the air quality dataset. Furthermore, Figure 3(b) shows our model outperforms GeoMAN-nt by a considerable margin since the temporal attention mechanism also enhances the long-term predictive performance.

• (b) Results on air quality.

Figure 4: Encoder length vs. metrics over the two datasets.

S27

S26

S0

S1

S23

S4

S6

S13

S11

S16

S17 S32

Target sensor Discussed sensor

(a) Air quality stations in Beijing

4.5 Case study

To further investigate our approach, we perform a case study over air quality dataset from 4:00 to 16:00 on Feb. 28, 2017. Figure 5(a) presents the distribution of air quality monitoring sensors throughout Beijing. For succinctness, we omit some sensors in this figure. We take the sensor S0 as an example to visualize the attention vectors for all encoder step t in Figure 5(b) and 5(c) respectively, where the attention weights are evolving across the encoder steps. Recall that the local weights semantically represent the relative importance of each local contributing series. According to the local weight matrix in Figure 5(b), the wind (from different directions) is an important factor affecting the concatenation of PM2.5, especially the local wind speed from the southeast. We also find that the temperature is closely related to the readings of PM2.5 at S0 since humans in northern China consume fuel for heating in the winter. On the contrary, the humidity and NO2 have no obvious effect on the concentration of PM2.5 at this moment. All these facts demonstrate that the local spatial attention successfully captures the correlation between the local features and target series. Then, we discuss the complex correlations between different sensors. As shown in Figure 5(c), for remote sensors (e.g., S16, S26 and S27), their attention weights are usually lower than that of nearby ones (e.g., S1 and S6). During this time, there was a strong wind blowing from the southeast. That is the reason why S11 and S13 are far away from S0 but have strong impacts on S0. Another interesting observation is that the attention weight of S23 increases by the time step because of the increasing southwest wind at that sensor. According to this case study, our method is not only effective but can also be easily interpreted. Due to the page limitation, we do not further present the weights in the temporal attention mechanism.

5 RelatedWork

Geo-sensory Time Series Prediction. Autoregression-based models (e.g., ARIMA and VAR) are widely used in time series prediction. Compared to traditional multivariate time series, geo-sensory time series has its own characteristics,

S3

e.g., spatial correlation. Recently, cross-domain fusion-based methods showed superiority in many spatio-temporal applications. [Zheng et al., 2015] forecast real-time air quality by fusing the past readings and the spatial factors. To predict the water quality, [Liu et al., 2016b] proposed a multi-task multi-view learning model, which jointly captures the local information as well as global information of each sensor.

Deep Learning for Spatio-Temporal Data. Recurrent neural networks (RNNs) become popular due to their success in sequence learning [Sutskever et al., 2014]. In particular, the incorporation of long short-term memory (LSTM) or gated recurrent unit (GRU) [Cho et al., 2014b] enables RNNs to learn long-term temporal dependency. However, these works can only capture temporal dependency in time series, which ignore the unique characteristics of geo-sensory data, e.g., spatial correlation. To overcome this problem, [Lv et al., 2015] first proposed a deep learning approach to extract the latent traffic flow feature representation, such as the nonlinear spatial and temporal correlations from the traffic data. Currently, [Zhang et al., 2017] developed a residual network considering both temporal and spatial dependencies to forecast the citywide crowd flow, a kind of geo-sensory time series. Attention Mechanism. Recently, attention mechanisms became popular due to its success in general sequence-tosequence problems. [Bahdanau et al., 2014] first introduced a general attention model that did not assume a monotonic alignment. Later, researchers developed a number of multilevel attention-based models to select the relevant features and encoder hidden states in different applications [Wang et al., 2016; Yu et al., 2017]. To forecast the time series, [Qin et al., 2017] proposed a dual-stage attention-based recurrent neural network (DA-RNN) to select the relevant driving series at each time interval. However, DA-RNN is not suitable for geo-sensory time series forecasting and the reason is detailed in Section 4.4. Thus, we propose two spatial different attention mechanisms to capture the dynamic inter-sensor correlations. To the best of our knowledge, no prior work studies our problem via an attention-based deep learning approach.

6 Conclusion and FutureWork

In this paper, we propose a novel multi-level attention-based network for predicting the geo-sensory time series based on heterogeneous data frommultiple domains. In the first level, local and global spatial attention mechanisms are applied to capture the dynamic inter-sensor correlations in geo-sensory data. In the second level, we employ a temporal attention to adaptively select the relevant time step to make predictions. Moreover, our model considers the effects of external factors by using a general fusion module. We evaluate our model on two types of geo-sensory datasets and the experiments show that our model achieves the best performance against 9 baselines in terms of the two metrics (RMSE and MAE) simultaneously. In addition, we visualize the attention weights to show the interpretation of our approach.

In the future, we will extend our method to solve the problem of long-term prediction. Moreover, we will explore the high-quality inference of geo-sensory time series through a limited number of sensors in real-world applications.

Acknowledgments

We thank Prof. Shuming Liu and Yipeng Wu from Tsinghua University for sourcing the water quality dataset in this study. This work was supported by the National Natural Science Foundation of China Grant No. 61672399, No. U1609217, and 973 Program, No. 2015CB352400.