Cross-City Transfer Learning for Deep Spatio-Temporal Prediction

1 Introduction

Spatio-temporal prediction covers a broad scope of applications in urban computing [Zheng et al., 2014], such as traffic and air quality prediction. Recently, with the development of big data techniques, deep learning becomes popular in spatio-temporal prediction, e.g. crowd flow, taxi demand, precipitation predictions, and achieves state-of-the-art performance [Ke et al., 2017; Shi et al., 2015; Zhang et al., 2017; Zhang et al., 2016]. However, the city development levels are quite unbalanced, so that many cities cannot benefit from such achievements due to data scarcity. Hence, how to help data-scarce cities also obtain benefits from the recent technique breakthroughs like deep leaning, becomes an important research issue, while it is still under-investigated up to date.

To tackle this problem, in this paper, we propose a new cross-city transfer learning method for deep spatio-temporal prediction tasks, called RegionTrans. The objective of RegionTrans is to predict a certain type of service data (e.g., crowd flow) in a data-scarce city (target city) by transferring knowledge learned from a data-rich city (source city). The principal idea of RegionTrans is to find inter-city region pairs that share similar patterns and then use such region pairs as proxies to efficiently transfer knowledge from the source city to the target.

This research is partially supported by Hong Kong ITF project no. ITS/391/15FX, and the collaboration project at HKUST-DiDi Joint Lab.

In literature, existing deep learning approaches are often designed to predict citywide phenomenon as a whole [Zhang et al., 2017; Zhang et al., 2016], and thus it is hard to enable region-level knowledge transfer. To this end, rather than adopting the existing deep neural networks for citywide spatio-temporal prediction, e.g. ST-ResNet [Zhang et al., 2017], we propose a novel deep transfer learning method. First, we design a region matching function to link each target city region to a similar source region based on the short period of service data or correlated auxiliary data if applicable. Then, in our proposed network structure, to catch the spatio-temporal patterns hidden in the service data, ConvLSTM layers [Shi et al., 2015] are firstly stacked. Afterward, to encode region representation, we newly add a Conv2D layer with 1×1 filter, which is the key and fundamental component of our network to make region-level transfer feasible. Finally, the discrepancy between region representations of the intercity similar regions is minimized during the network parameter learning for the target city, so as to enable region-level cross-city knowledge transfer. With crowd flow prediction as a showcase [Zhang et al., 2017], we verify the feasibility and effectiveness of RegionTrans.

Briefly, this paper has the following contributions.

• (i) To the best of our knowledge, this is the first work to facilitate deep spatio-temporal prediction in a data-scarce target city by transferring knowledge from a data-rich source city.
• (ii) We propose a novel deep transfer learning method RegionTrans for spatio-temporal prediction tasks by regionlevel cross-city transfer. RegionTrans first computes inter-city region similarities, and then stacks ConvLSTM and Conv2D (1 × 1 filter) layers to extract region-level representations reflecting spatio-temporal patterns. Finally, the discrepancy of the representations of inter-city similar regions is minimized so as to facilitate region-level cross-city knowledge transfer.
• (iii) With crowd flow prediction as a showcase, our experiment shows that RegionTrans can reduce up to 10.7% predic-

tion error compared to fine-tuned state-of-the-art deep spatiotemporal prediction methods.

2 RelatedWork

Spatio-Temporal Prediction is a fundamental problem in urban computing [Zheng et al., 2014]. Recently, deep learning is adopted in spatio-temporal prediction tasks and becomes the state-of-the-art solution when there exists a rich history of data. Various deep models have been used, e.g., CNN [Zhang et al., 2016], ResNet [Zhang et al., 2017], and ConvLSTM [Ke et al., 2017; Shi et al., 2015; Yao et al., 2018]. Compared to these works, the difference of our work lies in both objective and method. We aim to apply deep learning to a target city with a short period of service data, and thus propose RegionTrans to effectively transfer knowledge from a data-rich source city to the target city.

Transfer Learning addresses the machine learning problem when labeled training data is scarce [Pan and Yang, 2010]. In urban computing, data scarcity problem often exists when the targeted service or infrastructure is new. There are generally two strategies to deal with urban data scarcity. The first is using auxiliary data of the target city to help build the targeted application. Examples include using temperature to infer humidity and vice versa [Wang et al., 2017], and leveraging the taxi GPS traces to detect ridesharing cars [Wang et al., 2019]. The second is to find a source city with adequate data to transfer knowledge. Guo et al. design a crosscity transfer learning framework with collaborative filtering and AutoEncoder to conduct chain store site recommendation [Guo et al., 2018]. As our problem is prediction rather than recommendation, the method in [Guo et al., 2018] cannot be applied. Another relevant work is [Wei et al., 2016], which proposes a cross-city transfer learning algorithm FLORAL to predict air quality category. There are two difficulties to apply FLORAL to our task: (1) many spatio-temporal prediction tasks are regression but FLORAL is designed for classification; (2) FLORAL is not designed for deep learning. As far as we know, RegionTrans is the first cross-city transfer learning framework for deep spatio-temporal prediction.

3 ProblemFormulation

Definition 1. Region. [Zhang et al., 2016] A city D is partitioned into WD × HD equal-size grids (e.g., 1km × 1km). Each grid is called a region, denoted as r. We use r[i,j] to represent a city region whose coordinate is [i, j]. The whole set of regions in a city D is denoted as CD.

Definition 2. Urban Image Time Series. We denote the set of data time-stamps of a city D as:

where TD is the number of time-stamps and tc is the current/last time-stamp. For brevity, we consider equallength time-stamp (e.g., one-hour) as in the previous research [Zhang et al., 2017; Zhang et al., 2016]. For a specific timestamp t, we have an urban image It,D with WD ×HD pixels where each pixel represents certain data of a corresponding region (Def. 1),

Then, we define an urban image time series ID as follows:

In reality, a variety of urban data can be modeled as the above urban image time series, such as crowd flow, weather condition, air quality, etc.

Definition 3. Service Spatio-temporal Data. Service data is the targeted type of data to predict. We define the service spatio-temporal data as the urban image time series SD storing the service data :

where sr,t is the service data of region r at time-stamp t.

In this paper, the target city D suffers from the service data scarcity, while the source city D′ has rich service data, i.e., |TD| ≪ |TD′ |. With this in mind, we formulate the problem.

Problem of Cross-City Spatio-temporal Prediction. Given the little service data in target city D and rich service data in source city D′ , we aim to learn a function f to predict the citywide service data in the target city D at the next time-stamp tc + 1:

error metric may be mean absolute error, root mean squared error, etc., according to the real application requirement.

Example. Crowd Flow Prediction. We use crowd flow prediction [Zhang et al., 2017; Zhang et al., 2016] as an example to illustrate the above problem concretely. The service data SD is thus crowd inflow or outflow. The source city crowd flow records may last for several years (TD′ ), but the target city may have only a few days (TD) as the service is just started. It is worth noting that external context factors, such as weather and workday/weekend, are also important in crowd flow prediction [Zhang et al., 2017]. Later we will show that our proposed method is easy to add the external features extracted from context factors.

4 RegionTrans

4.1 Overview

Fig. 1 gives an overview of the RegionTrans framework. In brief, RegionTrans consists of three novel components.

1. Inter-city similar-region matching. We propose two ways to match region pairs between source and target cities. The first one is directly using the short period of service data in the target city to find the similar region in the source city. However, sometimes directly calculating the similarity between a source region and a target region using the short service data may not yield robust results. Suppose that the service data is crowd flow, the target city only has one day of crowd flow history and it happened to be a rainy day, but it rarely rains in the source city. Apparently, using such crowd flow data of the source and target city to compute inter-city region

similarity is inadequate. Hence, we propose a second method to rely on auxiliary data (if applicable) to find more robust inter-city similar region pairs. For example, widely-existing social check-ins may be the auxiliary data to indicate the crowd flow dynamics and thus help region matching.

1. Deep spatio-temporal neural network with region representations. Existing literature has proposed a few deep models for predicting citywide crowd flow [Zhang et al., 2016; Zhang et al., 2017]. However, these models usually predict citywide crowd flow as a whole, and thus are hard to incorporate region similarity information for transfer learning. Therefore, we propose a new deep spatio-temporal neural network structure, in which a 'region-representation' layer is dedicatedly designed to preserve region-level features. Based on this neural network, we then learn a source city spatio-temporal prediction model from its long historical record of service data and corresponding contexts (e.g., weather). This source city model will be later used in transfer learning for building the target city model.
2. Region-based cross-city network parameter optimization. Based on the deep model with region representations, we propose a cross-city network parameter optimization algorithm to learn the crowd flow prediction model for the target city, considering the source city model, the inter-city similar-region pairs, and the short period of crowd flow data of the target city.

4.2 Inter-city Similar-regionMatching

The first step of RegionTrans is to find a matching function M : CD → CD′ to map each region of the target city D to a certain region of the source city D′ . The objective is to find the source region having the similar spatio-temporal pattern with the target region. We propose two strategies to findM.

Matching with a Short Period of Service Data

While the target city has only a little service data, this could still provide hints to build M. We focus on the time span when both source and target cities have service data (i.e., TD), then calculate the correlations (e.g., Pearson coefficient) between each target region and source region with the corresponding service data. Finally, for each target region, we choose the source region with the largest correlation value. Formally,

Matching with a Long Period of Auxiliary Data

As there is little service data in the target city, the above service-data-based correlation similarity between a source region and a target region may not be very reliable. In reality, sometimes we can find another openly-accessible auxiliary data that correlates with the service data, which may help calculate the inter-city region similarity more robustly. For example, to predict crowd flow, social media check-ins can be a useful proxy according to literature [Yang et al., 2016]. That is, instead of the short period of crowd flow data, we use the long period of openly available check-in data to build the correlation between two regions.

where a is the auxiliary data (e.g., check-in number) lasting for a long period TA(|TA| ≫ |TD|).

Example: Check-in as Auxiliary for Crowd Flow

For a certain region r ∈ C in a city, we can model the check-in representation according to its hourly check-in counts in workday and weekend/holiday as follows:

where chi is the average check-in counts in r at ith hour in workday of the whole check-in historical record; ch′i is the hourly average check-in counts in weekend/holiday.

To verify whether the similarity between check-in representations can actually reflect the similarity of crowd flow dynamics, we conduct an analysis with bikesharing data in Washington D.C. and Chicago during 2015-2016. Here, we measure the crowd flow similarity between two regions by first counting the hourly bike inflow/outflow counts and then use the Pearson correlation coefficient. Given a randomly selected Chicago region r∗ , Fig. 2 plots both the check-in similarity (y axis) and crowd flow similarity (x axis) of every D.C. region and the selected Chicago region r∗ . As expected, the D.C. regions with higher check-in similarities tend to hold higher crowd flow similarities.

4.3 Deep Spatio-Temporal Neural Network with Region Representations

Existing deep spatio-temporal models often take the whole city data for end-to-end prediction, e.g., ST-ResNet [Zhang et al., 2017], which cannot be used for region-level transfer. Therefore, we design a network for spatio-temporal prediction with region representations, as shown in Fig. 3 (a).

First we illustrate the input and output of the network 1 :

k ∈ N+ : the length of the input time series

Our network objective is to minimize the squared error be-

1 For clarity, we omit the subscript D in notations as all the notations mentioned in this section is in city D.

Then, our spatio-temporal network can be formulated as:

(10)

ConvLSTM layers are the basic components for our network to learn spatio-temporal patterns [Shi et al., 2015]. We first use a set of stacked ConvLSTM layers to construct regionlevel hidden representation Xrep t ∈ RW×H×Lr (we will elaborate why this can be seen as region-level representation soon). After getting Xrep t , we incorporate the external context factors into the network structure. External context factors are defined as Xext t ∈ RW×H×Le , which is a feature vector of length Le on each region (e.g., weather, temperature, weekday/holiday one-hot encoding [Zhang et al., 2017]). By concatenating Xrep t and Xext t to form a representation ∈ RW×H×(Lr+Le) , we employ several convolution 2D layers with 1×1 filters (Conv2D1×1 [Lin et al., 2014]) to predict the next-time-stamp service data ˜ Yt ∈ RW×H .

As visualized by Fig. 3 (b), Conv2D1×1 will produce spatio-invariant results, which means hidden vector Xrep t [w, h, :] and prediction ˜ Yt[w, h] represent the spatiotemporal representation and prediction result of region r[w,h], respectively. Compared with existing end-to-end citywide deep spatio-temporal prediction models [Zhang et al., 2017; Zhang et al., 2016] without such region-level hidden representations, our network design has the following advantages for transfer learning:

1. Fine-grained region-level transfer. With existing methods which consider the data of a city as a whole for prediction, we can only transfer the knowledge from the whole source city to the target (e.g., through finetuning). If two cities are not similar in general, the transfer performance may be poor. As our network incorporates region representation, we can make fine-grained knowledge transfer based on region similarity (the detailed algorithm in the next sub-section). As long as we can find similar region pairs between cities, the effective transfer may be conducted.
2. Transfer between cities with different sizes. Since our neural network structure can be seen from region view (Fig. 3 (b)), even if two cities have different sizes (i.e. W,H), it is possible to train a model on a source city and then transfer the learned network parameters to the target city at the region level. However, with end-toend network structures [Zhang et al., 2017; Zhang et al., 2016], if we want to transfer a learned model from the

source city to the target by fine-tuning, the two cities must be the same size.

4.4 Region-based Cross-city Network Parameter Optimization

With the proposed network structure, we train a deep model in the source city D′ with its rich spatio-temporal service data. We denote θD′ as the network parameters learned from the source city. Then, with θD′ as the pre-trained network parameters, we propose a region-based cross-city optimization algorithm to refine the network parameters on the target city D, considering a short period TD of the service data in the target city D and the inter-city region matching function M.

When refining the network parameter for the target city D, the first objective is to minimize prediction error on D:

Given the matching function M, the second objective is to minimize representation divergence between matched region pairs. More specifically, for each time-stamp t ∈ TD, we try to minimize the squared error between the network hidden representations of the target region and its matched source region. Formally, the second objective is as follows:

where xrep r,t is the hidden representation of the target region r when the last input time-stamp is t; xrep r∗ ,t is the representation of the matched source region r∗ ; ρr,r∗ is the correlation value calculated between region pairs (Sec. 4.2), so that more similar pair will be assigned with larger weights in the optimization. Then, combining the two objectives leads to the following optimization process:

where w is the weight to trade off between minimizing the representation discrepancy or minimizing the prediction error. Then, we can use state-of-the-art network parameter learning algorithms, such as SGD and ADAM, to obtain the network parameter θD for the target city D according to Eq. 14 (the network parameter θD′ learned in the source city D′ is used as the initialization values). The detailed pseudocode of the optimization process is summarized in Alg. 1.

5 Experiment: Crowd Flow Prediction

We use crowd flow prediction as a case of spatio-temporal prediction tasks to evaluate RegionTrans.

5.1 Settings

Datasets. Following previous studies on crowd flow [Hoang et al., 2016; Zhang et al., 2017; Zhang et al., 2016], we use bike flow data for evaluation. Three bike flow datasets Algorithm 1 Region-based cross-city network parameter optimization

Input:

θD′ : Pre-trained network parameters on source city with a long period of service data

TRD: target city training data

TRD′ : source city training data

M: inter-city similar-region matching function

Output:

θD: network parameters for the target city

1: Initialize network parameters: θ ← θD′ 2: epoch← 0 3: while epoch ≤ MAX EPOCH do 4: for t ∈ TD do 5: Get {Xt, Yt} ∈ TRD 6: Get corresponding {X′t , Y ′ t } ∈ TRD′ 7: for r ∈ CD do 8: r∗ ←M(r) (note that r∗ ∈ CD′ ) 9: xrep r∗ ,t ← region representation with input X′t for source r∗ 10: xrep r,t ← region representation with input Xt for target r 11: end for 12: end for 13:

14: epoch ++

• 15: end while
• 16: θD ← θ
• 17: return θD

collected from Washington D.C., Chicago, and New York City are used. Each dataset covers a two-year period (2015-2016). In all the cities, the center area of 20km × 20km are selected. The area is split to 20 × 20 regions (each region is 1km× 1km). For each evaluation scenario, we choose one city as the source city and another as the target. We assume that the source city has all its historical crowd flow data, but only limited crowd flow data exists in the target city (e.g., one day). The last two-month data is chosen for testing.

Metric. The evaluation metric is root mean square error (RMSE). Same as [Zhang et al., 2017], the reported RMSE is the average RMSE of inflow and outflow.

Network Implementation. Our network structure implemented in the experiment has two layers of ConvLSTMwith 5 × 5 filters and 32 hidden states, to generate Xrep t ∈ R20×20×32 . With Xrep t as the input, there is one layer of Conv2D1×1 with 32 hidden states, followed by another layer of Conv2D1×1 linking to the output crowd flow prediction. For the external context factors, e.g., temperature, wind speed, weather, and day type, we use the same feature extraction method as [Zhang et al., 2017] and obtain an external feature vector with a length of 28. We also need to set w in Eq. 14 to balance the optimization trade-off between representation difference and prediction error. We set w to 0.75 as the default value. ADAM is used as the optimization algorithm [Kingma and Ba, 2015].

5.2 Methods

For RegionTrans, we implement two variants:

• RegionTrans (S-Match): learning the inter-city region matching function M only by the short period of the target city Service data, i.e., crowd flow.
• RegionTrans (A-Match): learning the inter-city region matching functionMby the long period of the Auxiliary data, i.e., Foursquare check-in data. We use one-year check-in data as the auxiliary data since it is a useful indication of crowd flow [Yang et al., 2016]. Note that we have collected check-in data from D.C. and Chicago, so RegionTrans (A-Match) is available for the knowledge transfer between them.

We compare RegionTrans with two types of baselines. The first type only uses the short crowd data history of target city:

• ARIMA: a widely-used time series prediction method in statistics [Hyndman and Athanasopoulos, 2014].
• DeepST [Zhang et al., 2016]: a deep spatio-temporal neural network based on convolutional network. The complete DeepST model has three components: closeness, period, and trend. But the period and trend components can only be activated if the training data last for more than one day and seven days, respectively. Therefore, if the target city does not have enough data, we have to deactivate the corresponding components.
• ST-ResNet [Zhang et al., 2017]: a deep spatio-temporal network based on residual network [He et al., 2016].

The second type trains a deep model on the source city data, and fine-tune it with the target city data:

• DeepST (FT): fine-tuned DeepST.
• ST-ResNet (FT): fine-tuned ST-ResNet.

As mentioned in Sec. 4.3, DeepST and ST-ResNet predict the city crowd flow as a whole, and thus we cannot fine tune their models between two cities of different sizes. Therefore, to make the comparison possible, our experiment selects the same area size for two cities. Note that RegionTrans is able to transfer knowledge between two cities of different sizes, and thus is more flexible.

5.3 Results

Table 1 shows our results for D.C. ⇌ Chicago and D.C. ⇌ NYC. RegionTrans can consistently outperform the best baseline, where the largest improvement is reducing RMSE by up to 10.7%. In particular, when the service data of the target city is smaller, the improvement of RegionTrans is usually more significant. This indicates that the inter-city similar-region pairs are valuable for transfer learning especially when target data is extremely scarce. Between two RegionTrans variants, RegionTrans (A-Match) is better, as shown in D.C. ⇌ Chicago. This implies that if an appropriate type of auxiliary data exists, it is possible to build a better inter-city region matching than only using the short period of the service data. If the auxiliary data is unavailable, using the limited period of service data for region matching can still lead to the competitive variant RegionTrans (S-Match), which beats all the baselines significantly.

We also train DeepST with the full training data in Chicago and D.C., respectively, leading to RMSE of 0.521 and 0.526, which could be seen as the performance upper bound of our experiment cases. Compared to RegionTrans (DC→Chicago, A-Match, 3-day) result of 0.576, the full Chicago model reduces 9.6% of RMSE; the full DC model is better than RegionTrans (Chicago→DC, A-Match, 3-day) by reducing 9.5% of RMSE. In comparison, the best baseline DeepST (FT, 3-day) is worse than the full model by 14.7% and 15.0% in Chicago and DC, respectively. Hence, RegionTrans can reduce the gap between the best baseline and the upper bound full model by ∼ 1/3, which is a significant improvement.

Another important observation is that RegionTrans is more robust when transferring knowledge between two dissimilar cities than baselines. Between the three cities in the experiment, D.C. and Chicago are similar in population, while NYC has a much larger population. This indicates that the knowledge transfer between D.C. ⇌ Chicago may be easier, while D.C. ⇌ NYC could be harder. Our results also verify this as DeepST and ST-ResNet get large improvement by fine-tuning in D.C. ⇌ Chicago; but in D.C. → NYC, negative transfer appears for the fine-tuned DeepST and ST-ResNet, leading to even worse performance than ARIMA, indicating that directly transferring the whole city knowledge from D.C. to NYC is ineffective. In comparison, RegionTrans consistently achieves lower error than all the baselines, verifying that the knowledge from D.C. can still be effectively transferred to NYC. The primary reason that RegionTrans can avoid negative transfer is that although D.C. and NYC are dissimilar in general, we can still find inter-city region pairs with similar spatio-temporal patterns (e.g., central business district).

Tuning w. We tune w in Eq. 14 to see how it will affect the performance. The larger w is, the higher weight is put on minimizing the similar-region representation difference. Fig. 4 shows the results. If we set w = 0 (i.e., direct fine tuning without region matching), the performance is significantly worse than when w > 0, by incurring up to 5% higher error. This highlights the effectiveness of our proposed intercity similar-region matching scheme in cross-city knowledge transfer. For other settings of w > 0, the performance difference is minor. A larger w performs slightly better for a very short period of target city crowd flow data, e.g., one day.

Computation time. We use a server with Intel Xeon CPU E5-2650L, 128 GB RAM, and Nvidia Tesla M60 GPU. We implement RegionTrans with TensorFlow (CentOS). Training the source city model on two-year data needs ∼20 minutes, and the transfer learning for the target city model costs ∼50 and ∼100 minutes for 1 and 3-day data, respectively. This running time efficiency is acceptable in real-life deployments.

6 Conclusion

To address the data scarcity issue in urban spatio-temporal prediction tasks, this paper proposes a novel cross-city deep transfer learning method, called RegionTrans. Note that while this paper focuses on cross-city transfer, the key idea of RegionTrans can be flexibly leveraged with other spatial granularities, e.g., district-level or country-level transfer. In the future, we will study how effective transfer can be done on other recent spatio-temporal models such as graph convolutional networks [Geng et al., 2019; Li et al., 2018].