Graph WaveNet for Deep Spatial-Temporal Graph Modeling

1 Introduction

Spatial-temporal graph modeling has received increasing attention with the advance of graph neural networks. It aims to model the dynamic node-level inputs by assuming interdependency between connected nodes, as demonstrated by Figure 1. Spatial-temporal graph modeling has wide applications in solving complex system problems such as traffic speed forecasting [Li et al., 2018b], taxi demand prediction [Yao et al., 2018], human action recognition [Yan et al., 2018], and driver maneuver anticipation [Jain et al., 2016]. For a concrete example, in traffic speed forecasting, speed sensors on roads of a city form a graph where the edge weights are judged by two nodes' Euclidean distance. As the traffic congestion on one road could cause lower traffic speed on its incoming roads, it is natural to consider the underlying graph structure of the traffic system as the prior knowledge of inter-dependency relationships among nodes when modeling time series data of the traffic speed on each road.

∗Corresponding Author.

A basic assumption behind spatial-temporal graph modeling is that a node's future information is conditioned on its historical information as well as its neighbors' historical information. Therefore how to capture spatial and temporal dependencies simultaneously becomes a primary challenge. Recent studies on spatial-temporal graph modeling mainly follow two directions. They either integrate graph convolution networks (GCN) into recurrent neural networks (RNN) [Seo et al., 2018; Li et al., 2018b] or into convolution neural networks (CNN) [Yu et al., 2018; Yan et al., 2018]. While having shown the effectiveness of introducing the graph structure of data into a model, these approaches face two major shortcomings.

First, these studies assume the graph structure of data reflects the genuine dependency relationships among nodes. However, there are circumstances when a connection does not entail the inter-dependency relationship between two nodes and when the inter-dependency relationship between two nodes exists but a connection is missing. To give each circumstance an example, let us consider a recommendation system. In the first case, two users are connected, but they may have distinct preferences over products. In the second case, two users may share a similar preference, but they are not linked together. Zhang et al. [2018] used attention mechanisms to address the first circumstance by adjusting the dependency weight between two connected nodes, but they failed to consider the second circumstance.

Second, current studies for spatial-temporal graph modeling are ineffective to learn temporal dependencies. RNNbased approaches suffer from time-consuming iterative propagation and gradient explosion/vanishing for capturing longrange sequences [Seo et al., 2018; Li et al., 2018b; Zhang et al., 2018]. On the contrary, CNN-based approaches enjoy the advantages of parallel computing, stable gradients and low memory requirement [Yu et al., 2018; Yan et al., 2018]. However, these works need to use many layers in order to capture very long sequences because they adopt standard 1D convolution whose receptive field size grows linearly with an increase in the number of hidden layers.

In this work, we present a CNN-based method named Graph WaveNet, which addresses the two shortcomings we have aforementioned. We propose a graph convolution layer in which a self-adaptive adjacency matrix can be learned from the data through an end-to-end supervised training. In this way, the self-adaptive adjacency matrix preserves hidden spatial dependencies. Motivated by WaveNet [Oord et al., 2016], we adopt stacked dilated casual convolutions to capture temporal dependencies. The receptive field size of stacked dilated casual convolution networks grows exponentially with an increase in the number of hidden layers. With the support of stacked dilated casual convolutions, Graph WaveNet is able to handle spatial-temporal graph data with long-range temporal sequences efficiently and effectively. The main contributions of this work are as follows:

• We construct a self-adaptive adjacency matrix which preserves hidden spatial dependencies. Our proposed self-adaptive adjacency matrix is able to uncover unseen graph structures automatically from the data without any guidance of prior knowledge. Experiments validate that our method improves the results when spatial dependencies are known to exist but are not provided.
• We present an effective and efficient framework to capture spatial-temporal dependencies simultaneously. The core idea is to assemble our proposed graph convolution with dilated casual convolution in a way that each graph convolution layer tackles spatial dependencies of nodes' information extracted by dilated casual convolution layers at different granular levels.
• We evaluate our proposed model on traffic datasets and achieve state-of-the-art results with low computation costs. The source codes of Graph WaveNet are publicly available from https://github.com/ nnzhan/Graph-WaveNet.

2 RelatedWorks

2.1 Graph Convolution Networks

Graph convolution networks are building blocks for learning graph-structured data [Wu et al., 2019]. They are widely applied in domains such as node embedding [Pan et al., 2018], node classification [Kipf and Welling, 2017], graph classification [Ying et al., 2018], link prediction [Zhang and Chen, 2018] and node clustering [Wang et al., 2017]. There are two mainstreams of graph convolution networks, the spectralbased approaches and the spatial-based approaches. Spectralbased approaches smooth a node's input signals using graph spectral filters [Bruna et al., 2014; Defferrard et al., 2016; Kipf and Welling, 2017]. Spatial-based approaches extract a node's high-level representation by aggregating feature information from neighborhoods [Atwood and Towsley, 2016; Gilmer et al., 2017; Hamilton et al., 2017]. In these approaches, the adjacency matrix is considered as prior knowledge and is fixed throughout training. Monti et al. [2017] learned the weight of a node's neighbor through Gaussian kernels. Velickovic et al. [2017] updated the weight of a node's neighbor via attention mechanisms. Liu et al. [2019] proposed an adaptive path layer to explore the breadth and depth of a node's neighborhood. Although these methods assume the contribution of each neighbor to the central node is different and need to be learned, they still rely on a predefined graph structure. Li et al. [2018a] adopted distance metrics to adaptively learn a graph's adjacency matrix for graph classification problems. This generated adjacency matrix is conditioned on nodes' inputs. As inputs of a spatialtemporal graph are dynamic, their method is unstable for spatial-temporal graph modeling.

2.2 Spatial-temporal Graph Networks

The majority of Spatial-temporal Graph Networks follows two directions, namely, RNN-based and CNN-based approaches. One of the early RNN-based methods captured spatial-temporal dependencies by filtering inputs and hidden states passed to a recurrent unit using graph convolution [Seo et al., 2018]. Later works adopted different strategies such as diffusion convolution [Li et al., 2018b] and attention mechanisms [Zhang et al., 2018] to improve model performance. Another parallel work used node-level RNNs and edge-level RNNs to handle different aspects of temporal information [Jain et al., 2016]. The main drawbacks of RNN-based approaches are that it becomes inefficient for long sequences and its gradients are more likely to explode when they are combined with graph convolution networks. CNN-based approaches combine a graph convolution with a standard 1D convolution [Yu et al., 2018; Yan et al., 2018]. While being computationally efficient, these two approaches have to stack many layers or use global pooling to expand the receptive field of a neural network model.

3 Methodology

In this section, we first give the mathematical definition of the problem we are addressing in this paper. Next, we describe two building blocks of our framework, the graph convolution layer (GCN) and the temporal convolution layer (TCN). They work together to capture the spatial-temporal dependencies. Finally, we outline the architecture of our framework.

3.1 ProblemDefinition

A graph is represented by G = (V,E) where V is the set of nodes and E is the set of edges. The adjacency matrix derived from a graph is denoted by A ∈ RN×N . If vi, vj ∈ V and (vi, vj) ∈ E, then Aij is one otherwise it is zero. At each time step t, the graph G has a dynamic feature matrix X(t) ∈ RN×D . In this paper, the feature matrix is used interchangeably with graph signals. Given a graph G and its historical S step graph signals, our problem is to learn a function f which is able to forecast its next T step graph signals. The mapping relation is represented as follows

3.2 Graph Convolution Layer

Graph convolution is an essential operation to extract a node's features given its structural information. Kipf et al. [2017] proposed a first approximation of Chebyshev spectral filter [Defferrard et al., 2016]. From a spatial-based perspective, it smoothed a node's signal by aggregating and transforming its neighborhood information. The advantages of their method are that it is a compositional layer, its filter is localized in space, and it supports multi-dimensional inputs. Let ˜A ∈ RN×N denote the normalized adjacency matrix with self-loops, X ∈ RN×D denote the input signals , Z ∈ RN×M denote the output, and W ∈ RD×M denote the model parameter matrix, in [Kipf and Welling, 2017] the graph convolution layer is defined as

Li et al. [2018b] proposed a diffusion convolution layer which proves to be effective in spatial-temporal modeling. They modeled the diffusion process of graph signals with K finite steps. We generalize its diffusion convolution layer into the form of Equation 2, which results in,

where Pk represents the power series of the transition matrix. In the case of an undirected graph, P = A/rowsum(A). In the case of a directed graph, the diffusion process have two directions, the forward and backward directions, where the forward transition matrix Pf = A/rowsum(A) and the backward transition matrix Pb = AT/rowsum(AT). With the forward and the backward transition matrix, the diffusion graph convolution layer is written as

Self-adaptive Adjacency Matrix

In our work, we propose a self-adaptive adjacency matrix ˜Aadp. This self-adaptive adjacency matrix does not require any prior knowledge and is learned end-to-end through stochastic gradient descent. In doing so, we let the model discover hidden spatial dependencies by itself. We achieve this by randomly initializing two node embedding dictionaries with learnable parameters E1, E2 ∈ RN×c . We propose the self-adaptive adjacency matrix as

We name E1 as the source node embedding and E2 as the target node embedding. By multiplying E1 and E2, we derive the spatial dependency weights between the source nodes and the target nodes. We use the ReLU activation function to eliminate weak connections. The SoftMax function is applied to normalize the self-adaptive adjacency matrix. The normalized self-adaptive adjacency matrix, therefore, can be considered as the transition matrix of a hidden diffusion process. By combining pre-defined spatial dependencies and self-learned hidden graph dependencies, we propose the following graph convolution layer

When the graph structure is unavailable, we propose to use the self-adaptive adjacency matrix alone to capture hidden spatial dependencies, i.e.,

It is worth to note that our graph convolution falls into spatial-based approaches. Although we use graph signals interchangeably with node feature matrix for consistency, our graph convolution in Equation 7 indeed is interpreted as aggregating transformed feature information from different orders of neighborhoods.

3.3 Temporal Convolution Layer

We adopt the dilated causal convolution [Yu and Koltun, 2016] as our temporal convolution layer (TCN) to capture a node's temporal trends. Dilated causal convolution networks allow an exponentially large receptive field by increasing the layer depth. As opposed to RNN-based approaches, dilated casual convolution networks are able to handle longrange sequences properly in a non-recursive manner, which facilitates parallel computation and alleviates the gradient explosion problem. The dilated causal convolution preserves the temporal causal order by padding zeros to the inputs so that predictions made on the current time step only involve historical information. As a special case of standard 1Dconvolution, the dilated causal convolution operation slides over inputs by skipping values with a certain step, as illustrated by Figure 2. Mathematically, given a 1D sequence input x ∈ RT and a filter f ∈ RK , the dilated causal convolution operation of x with f at step t is represented as

where d is the dilation factor which controls the skipping distance. By stacking dilated causal convolution layers with dilation factors in an increasing order, the receptive field of a model grows exponentially. It enables dilated causal convolution networks to capture longer sequences with less layers, which saves computation resources.

Gated TCN. Gating mechanisms are critical in recurrent neural networks. They have been shown to be powerful to control information flow through layers for temporal convolution networks as well [Dauphin et al., 2017]. A simple Gated TCN only contains an output gate. Given the input X ∈ RN×D×S , it takes the form

where Θ1, Θ2, b and c are model parameters, ⊙ is the element-wise product, g(·) is an activation function of the outputs, and σ(·) is the sigmoid function which determines the ratio of information passed to the next layer. We adopt Gated TCN in our model to learn complex temporal dependencies. Although we empirically set the tangent hyperbolic function as the activation function g(·), other forms of Gated TCN can be easily fitted into our framework, such as an LSTM-like Gated TCN [Kalchbrenner et al., 2016].

3.4 Framework of GraphWaveNet

We present the framework of Graph WaveNet in Figure 3. It consists of stacked spatial-temporal layers and an output layer. A spatial-temporal layer is constructed by a graph convolution layer (GCN) and a gated temporal convolution layer (Gated TCN) which consists of two parallel temporal convolution layers (TCN-a and TCN-b). By stacking multiple spatial-temporal layers, Graph WaveNet is able to handle spatial dependencies at different temporal levels. For example, at the bottom layer, GCN receives short-term temporal information while at the top layer GCN tackles long-term temporal information. The inputs h to a graph convolution layer in practice are three-dimension tensors with size [N,C,L] where N is the number of nodes, and C is the hidden dimension, L is the sequence length. We apply the graph convolution layer to each of h[:, :, i] ∈ RN×C .

We choose to use mean absolute error (MAE) as the training objective of Graph WaveNet, which is defined by

Unlike previous works such as [Li et al., 2018b; Yu et al., 2018], our Graph WaveNet outputs ˆX(t+1):(t+T) as a whole rather than generating ˆX(t) recursively through T steps. It addresses the problem of inconsistency between training and testing due to the fact that a model learns to make predictions for one step during training and is expected to produce predictions for multiple steps during inference. To achieve this, we artificially design the receptive field size of Graph WaveNet equals to the sequence length of the inputs so that in the last spatial-temporal layer the temporal dimension of the outputs exactly equals to one. After that we set the number of output channels of the last layer as a factor of step length T to get our desired output dimension.

4 Experiments

We verify Graph WaveNet on two public traffic network datasets, METR-LA and PEMS-BAY released by Li et al. [2018b]. METR-LA records four months of statistics on traffic speed on 207 sensors on the highways of Los Angeles County. PEMS-BAY contains six months of traffic speed information on 325 sensors in the Bay area. We adopt the same data pre-processing procedures as in [Li et al., 2018b]. The readings of the sensors are aggregated into 5-minutes windows. The adjacency matrix of the nodes is constructed by road network distance with a thresholded Gaussian kernel [Shuman et al., 2012]. Z-score normalization is applied to inputs. The datasets are split in chronological order with 70% for training, 10% for validation and 20% for testing. Detailed dataset statistics are provided in Table 1.

4.1 Baselines

We compare Graph WaveNet with the following models.

• ARIMA. Auto-Regressive Integrated Moving Average model with Kalman filter [Li et al., 2018b].

• FC-LSTM Recurrent neural network with fully connected LSTM hidden units [Li et al., 2018b].
• WaveNet. A convolution network architecture for sequence data [Oord et al., 2016]. 70
• DCRNN. Diffusion convolution recurrent neural network [Li et al., 2018b], which combines graph convolution networks with recurrent neural networks in an encoder-decoder manner. 60 65 Value

55

• GGRU. Graph gated recurrent unit network [Zhang et al., 2018]. Recurrent-based approaches. GGRU uses attention mechanisms in graph convolution. 50

45

• STGCN. Spatial-temporal graph convolution network [Yu et al., 2018], which combines graph convolution with 1D convolution. 0 100

4.2 Experimental Setups

Our experiments are conducted under a computer environment with one Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz and one NVIDIA Titan Xp GPU card. To cover the input sequence length, we use eight layers of Graph WaveNet with a sequence of dilation factors 1, 2, 1, 2, 1, 2, 1, 2. We use Equation 4 as our graph convolution layer with a diffusion step K = 2. We randomly initialize node embeddings by a uniform distribution with a size of 10. We train our model using Adam optimizer with an initial learning rate of 0.001. Dropout with p=0.3 is applied to the outputs of the graph convolution layer. The evaluation metrics we choose include mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE). Missing values are excluded both from training and testing.

4.3 Experimental Results

Table 2 compares the performance of Graph WaveNet and baseline models for 15 minutes, 30 minutes and 60 minutes ahead prediction on METR-LA and PEMS-BAY datasets. Graph WaveNet obtains the superior results on both datasets.

Time

300 400 Figure 4: Comparison of prediction curves between WaveNet and Graph WaveNet for 60 minutes ahead prediction on a snapshot of the test data of METR-LA.

It outperforms temporal models including ARIMA, FCLSTM, and WaveNet by a large margin. Compared to other spatial-temporal models, Graph WaveNet surpasses the previous convolution-based approach STGCN significantly and excels recurrent-based approaches DCRNN and GGRU at the same time. In respect of the second best model GGRU, Graph WaveNet achieves small improvement over GGRU on the 15minute horizons; however, realizes bigger enhancement on the 60-minute horizons. Our architecture is more capable of detecting spatial dependencies at each temporal stage. GGRU uses recurrent architectures in which parameters of the GCN layer are shared across all recurrent units. In contrast, Graph WaveNet employs stacked spatial-temporal layers which contain separate GCN layers with different parameters. Therefore each GCN layer in Graph WaveNet is able to focus on its own range of temporal inputs.

We plot 60-minutes-ahead predicted values v.s real values of Graph WaveNet and WaveNet on a snapshot of the test data in Figure 4. It shows that Graph WaveNet generates more stable predictions than WaveNet. In particular, there is a red sharp spike produced by WaveNet, which deviates far from real values. On the contrary, the curve of Graph WaveNet goes in the middle of real values all the time.

200

Time WaveNet Graph WaveNet Real Value

8

Figure 5: The learned self-adaptive adjacency matrix.

Effect of the Self-Adaptive Adjacency Matrix

To verify the effectiveness of our proposed adaptive adjacency matrix, we conduct experiments with Graph WaveNet using five different adjacency matrix configurations. Table 3 shows the average score of MAE, RMSE, and MAPE over 12 prediction horizons. We find that the adaptive-only model works even better than the forward-only model with mean MAE. When the graph structure is unavailable, Graph WaveNet would still be able to realize a good performance. The forward-backward-adaptive model achieves the lowest scores on all three evaluationmetrics. It indicates that if graph structural information is given, adding the self-adaptive adjacency matrix could introduce new and useful information to the model. In Figure 5, we further investigate the learned self-adaptive adjacency matrix under the configuration of the forward-backward-adaptive model trained on the METR-LA dataset. According to Figure 5a, some columns have more high-value points than others such as column 9 in the left box compared to column 47 in the right box. It suggests that some nodes are influential to most nodes in a graph while other nodes have weaker impacts. Figure 5b confirms our observation. It can be seen that node 9 locates nearby the intersection of several main roads while node 47 lies in a single road.

Computation Time

We compare the computation cost of Graph WaveNet with DCRNN and STGCN on the METR-LA dataset in Table 4.

31

40

Table 4: The computation cost on the METR-LA dataset.

Graph WaveNet runs five times faster than DCRNN but two times slower than STGCN in training. For inference, we measure the total time cost of each model on the validation data. Graph WaveNet is the most efficient of all at the inference stage. This is because that Graph WaveNet generates 12 predictions in one run while DCRNN and STGCN have to produce the results conditioned on previous predictions.

5 Conclusion

In this paper, we present a novel model for spatial-temporal graph modeling. Our model captures spatial-temporal dependencies efficiently and effectively by combining graph convolution with dilated casual convolution. We propose an effective method to learn hidden spatial dependencies automatically from the data. This opens up a new direction in spatialtemporal graph modeling where the dependency structure of a system is unknown but needs to be discovered. On two public traffic network datasets, Graph WaveNet achieves state-ofthe-art results. In future work, we will study scalable methods to apply Graph WaveNet on large-scale datasets and explore approaches to learn dynamic spatial dependencies.

Acknowledgments

This research was funded by the Australian Government through the Australian Research Council (ARC) under grants 1) LP160100630 partnership with Australia Government Department of Health and 2) LP150100671 partnership with Australia Research Alliance for Children and Youth (ARACY) and Global Business College Australia (GBCA).