When Transfer Learning Meets Cross-City Urban Flow Prediction: Spatio-Temporal Adaptation Matters

1 Introduction

Urban flow prediction is indispensable for constructing smart cities and intelligent traffic systems (ITS). It is a typical spatio-temporal prediction task that aims to forecast future crowd flows over citywide regions based on historical observation data (e.g., taxi and bike trip records). Accurate urban flow prediction serves a broad scope of applications in transportation [Yuan et al., 2020], mobility mining [Mazimpaka and Timpf, 2016], and urban computing [Zheng, 2015]. Due to its great benefits, many studies are proposed [Jiang et al., 2021; Yuan and Li, 2021; Ye et al., 2020]. The essential task behind is to capture the spatio-temporal urban dependence for spatio-temporal data prediction.

Recently, inspired by the success of deep neural networks, deep learning based methods such as [Xingjian et al., 2015; Zhang et al., 2017; Li et al., 2018a; Yao et al., 2019b; Liang et al., 2019; Zhang et al., 2020; Yuan et al., 2021] are emerging and achieve remarkable performance. While the researchers apply increasingly compelling networks (varying from RNNs to CNNs to GNNs) to improve the accuracy, it simultaneously exposes unignorable problems. Specifically, the superior performance of these methods highly relies on large-scale training data such as massive vehicle trip records or even auxiliary weather information [Liao et al., 2018], which are usually inaccessible in real world. In reality, datascare issues commonly exist due to sparse positioning stations, data privacy principles, and low city development levels [Wang et al., 2019]. For instance, some cities like New York and Beijing have released taxi data for a couple of years, while others like Chengdu may only release a few days of data. Although travel platforms (e.g., Didi and Uber) may collect massive vehicle trips via navigation services, they cannot release the data under privacy issues [Tong et al., 2017]. As a result, existing approaches trained on data-rich cities (e.g., NYC and BJ) are hard to accommodate those cities with sparse data [Wang et al., 2018]. Correspondingly, their performance may degrade when the training samples decrease.

Targets. Instead of proposing yet another deep learning approach requiring a large volume of spatio-temporal data to further optimize the model accuracy, in this paper, we propose a transfer learning framework to perform urban flow prediction in a data-scarce target city via the knowledge transferred from a data-rich source city. The core task is to capture the shareable (common) spatio-temporal features between the source and target cities, in which case, the model trained on source city could be generalized to support target city.

Although recent studies [Wang et al., 2019; Yao et al., 2019a; Wang et al., 2021] have made some efforts, they mainly focus on static and spatial-only feature transfer while neglecting the dynamic correlations accompanying the time dimension. The dynamic here means that the interactions of two cities at different times are supposed to be different, as highlighted by red dotted rectangle in Fig. 1 (b). In view of this, the temporal dimension must be transferred with the spatial dimension. Motivated by Domain Adaptation [Long et al., 2015; Chen et al., 2019] (a type of transfer learning methods in Computer Vision) to learn domain-invariant features across two images, we borrow its idea to bridge source domain (city) and target domain (city) in isomorphic latent space considering spatial and temporal aspects to capture the domain/city invariant features for target city prediction.

However, it is a nontrivial endeavor to design such a domain adaptation approach due to the following challenges.

• C1: How to capture the spatial domain-invariant correlations between the source and target cities? As depicted in Fig. 1, the spatial correlations between two cities may vary across time. To learn sufficient features, a natural way is to use discrepancy reduction methods to explicitly compute the spatial distributions of two cities at each time step. However, in that case, massive spatial features (each time step corresponds to a feature collection) are yielded to be computed, which could result in expensive computational costs.
• C2: How to capture the temporal domain-invariant correlations between the source and target cities? Existing methods typically neglect the temporal correlations between source and target cities, which constrains themselves from temporal-aware feature transfer. The temporal dynamics can be viewed as a combination of multiple local temporal features corresponding to different spatial features. However, not all the local temporal features equally contribute to overall spatio-temporal feature transfer. To address this, a promising way is to focus more on those with high contributions to the domain discrepancy between the source and target cities.

With these challenges in mind, we propose a SpatioTemporal Adaptation Network called STAN, which consists of a spatial adversarial adaptation module (SAAM), a temporal attentive adaptation module (TAAM), and a prediction module (PM). Specifically, we design the SAAM to solve the first challenge. Inspired by GANs [Goodfellow et al., 2014], it distinguishes the spatial features from the source and target cities at each time step via adversarial classification to effciently capture spatio-transferable features in an efficient manner. To overcome the second challenge, we design the TAAM, which adopts attention mechanisms when performing domain discrepancy reduction. By doing it, the temporal dynamics that contribute more to the domain discrepancy will be paid more attention to, leading to more effective spatiotemporal feature transfer. Finally, under the guidance of PM, STAN enable transferring task-driven features for data-scare spatio-temporal urban flow prediction.

In summary, this paper makes the following contributions.

• Spatio-Temporal Transfer Learning. We propose a novel transfer learning framework to capture sufficient spatiotemporal transferable knowledge via domain adaptation. The framework enables cross-city urban flow prediction, which can effectively address data-scarce problems.
• Spatial Domain Adaptation. We design a spatial adversarial adaptation module to transfer spatial domaininvariant features while learning dynamic spatial correlations in an efficient adversarial-based classification way, successfully avoiding high computational costs.
• Temporal Domain Adaptation. We design a temporal attentive adaptation module, which simultaneously aligns temporal domains and encodes temporal dynamics into spatiotemporally-aware representations. Moreover, it utilizes attention mechanisms to capture critical temporal features for fine-grained transfer learning.
• We conduct comprehensive experiments to evaluate STAN. The results show that STAN i) outperforms both fine-tuned deep learning methods and state-of-the-art transfer learning methods; ii) exhibits superior robustness to data scarcity and parameter settings; and iii) shows remarkable model efficiency.

2 ProblemFormulation

Definition 1 (Region). Following previous works [Wang et al., 2019; Yao et al., 2019a], we spatially split a city c into a h × w (e.g., 16 × 16) grid image based on the longitude and latitude. Each grid is defined as a region ri,j that locates at ith row and jth column, and all the regions of the city form a region image Rc = {r1,1, ..., ri,j , ..., rh,w}.

Definition 2 (Inflow/Outflow). In urban context, each region ri,j during time interval t involves two types of flows, namely inflow and outflow, which can be computed based on the historical vehicle trip records/trajectories.

where T is a trajectory set, τ = {g1, ..., gt, ..., gn} is a trajectory of length n, and gt denotes t-th GPS location of τ .

Definition 3 (Urban Flow Image). Given region image R = {r1,1, ..., ri,j , ..., rh,w} of a city, we define the inflow and outflow during time interval t of all regions as a urban flow image donated as a matrix Xt ∈ Rh×w×2 .

Here, the in/outflow in each region can be replaced with any other spatio-temporal data such as pickup and weather data so that our proposed model can flexibly support spatio-temporal prediction tasks such as demand and environment forecasting.

Definition 4 (Urban Flow Image Time-Series). Given the latest time interval t and historical time range k, we denote the urban flow image time-series as X = {Xt-k+1, ...,Xt-1,Xt} ∈ Rk×h×w×2 , where X(i, j, t, ∗) denotes the inflow and outflow in region ri,j during t.

Problem Definition. Given the sufficient data in a source city S and insufficient data in a target city T (i.e., |X S | ≫ |X T |), transfer-based cross-city urban flow prediction aims to learn a function f to forecast the crowd flow in target city T of all regions during the next time interval t + 1:

Here, ˆ Y T t+1 is the predicted result while Y T t+1 is the ground truth. error can be measured using mean absolute error (MAE), root mean squared error (RMSE), etc.

3 Methodology

3.1 Framework Overview

Fig. 2 shows the framework of STAN, which contains three major modules MS, MT , and MP . STAN runs three modules jointly for spatio-temporal knowledge transfer predictive learning. Three losses including spatial domain loss Ls, temporal domain loss Lt and prediction loss Lp are aligned with three modules, respectively. A mixed loss function is further designed for model training. The detailed structure of each module is described in the following subsections.

3.2 Spatial Adversarial AdaptationModule

As shown in Fig. 2, SAAM is composed of spatial extractor Gse and spatial discriminator Gsd to learn spatial transferable features in an adversarial way.

Spatial Feature Extractor Gse. Following most of previous studies, we apply CNNs to extract the spatial features since we treat source/target cities like images. Specifically, given historical data of P days where each day contains k time intervals before the current time interval of the source city, the input urban flow XS = {X (S,1) ;X (S,2) ; ...;X (S,P)} = {X(S,1) t-k , ...,X (S,1) t ; X(S,2) t-k , ...,X (S,2) t ; ...; X(S,P ) t-k , ...,X (S,P) t ∈ RP×k×h×w×2 . Next, each X (S,i) (the crowd flow of source city at day i) is fed into a convolution layer as below:

According to Eq. 3, we can obtain the initial spatial feature set of the target city, i.e., Z S = {zS 1 , ..., zS P-1 , zS P } ∈

RP×k×N×D , where k denotes the channels of kernel, N is the dimension of the two-dimensional matrix after paving, and D is the output feature dimension. Similarly, we can obtain the spatial feature set of the target city, i.e., Z T = {zT 1 , ..., z T P-1 , zT P } ∈ RP×k×N×D in the same way.

Spatial Feature Discriminator Gsd. Based on the obtained Z S and Z T , we further label these P × k samples (i.e., spatial features) as 1 or 0, where 1 is assigned to source city data as real data while 0 is assigned to target city data as fake data. Then, as depicted in Fig. 2, we introduce a spatial feature discriminatory Gsd via fully connected (FC) layers. Specifically, we mix the labeled Z S and Z T as the input of Gsd. Gsd then distinguishes the features from the source or target city. Its gradient training function is defined as follows:

where Wd denotes the parameters of Gsd. zs (i) and zt (i) represent spatial feature of source and target cities, respectively. In terms of Gsd training, we add the gradient in Eq. 4 to get larger gradient for better classification performance. In contrast, Gse aims to mix the spatial features of two cities as close as possible, so it subtracts the gradient for training:

where We denotes the parameters of Gse.

Spatial Domain Loss Ls. Through the adversarial objectives, the discriminator Gsd is optimized to classify input spatial features into different domains, while the feature extractor Gse is optimized in the opposite direction. Then, we define the spatial domain loss function as:

BCELoss is widely used in classification tasks. The smaller the Ls, the closer the spatial distributions of two cities.

3.3 Temporal Attentive AdaptationModule

As depicted in Fig. 2, TAAMMt is designed for time-aware feature transfer as urban discrepancies are also reflected in the temporal dimension. Specifically, we first use the selfattention mechanisms to capture the domain weight, representing the impact of local temporal features of source and target cities. Then, the discrepancy-based domain adaptation method, i.e., MMD [Long et al., 2017] is employed to capture the global temporal features from source and target cities.

Self-attention Based Domain Weight. We first utilize LSTM for local temporal feature embedding as LSTM is originally designed to capture the correlations in time series. As observed, we feed Z S = {zS 1 , ..., zS P-1 , zS P } and Z T = {zT 1 , ..., z T P-1 , zT P } to P LSTMs. The length of each LSTM is k. After concating the output of each LSTM, we compute the initial local temporal feature set of source and target cities, i.e., tS = {tS1 , ..., tSP-1 , tSP } ∈ RP×k×D and tT = {tT1 , ..., tTP-1 , tTP } ∈ RP×k×D .

Based on local temporal features, we can compute the global temporal features by summing each local temporal feature. However, not all the local temporal features equally contribute to the global temporal feature, and thus the internal correlation between local temporal features matters. To address this, as shown in Fig. 2, Mt introduces self-attention layers to consider both local and global aspects.

Specifically, to perform feature adaptation between source and target cities, we design a domain weight to capture those important local temporal features while discarding those local temporal features that may negatively affect domain transfer. The weight α of self-attention is defined as:

where Q,K, V ∈ RP×P represent query vector, key vector and value vector, respectively. W Q,W k ,W V ∈ RD×k×P are random initialization parameter matrices, α ∈ RP×P is the weight matrix w.r.t. Q,K, V . Dk is the dimension of Q or V . Each local temporal feature can be calculated as:

where vj is j-th vector of V , αpj represents correlation weight of j-th local feature to p-th local feature. Then, an initial global temporal feature of source city is:

The initial global temporal feature of the target city, i.e., tT , can be obtained in the same way. Next, we consider reducing the feature discrepancies between the source and target cities in terms of local-to-local and global-to-global dimensions. Specifically, we utilize MMD that is widely used as a distribution distance measure.

where F S = f S ∗ and F T = f T ∗ denote the features of two cities, respectively. n and m denote feature dimensions. E(·) represents kernel function such as Gaussian kernel, and H represents Hilbert space. According to Eq. 10, the local temporal feature discrepancy of day p is:

where M denotes feature dimensions. The global temporal feature dg is calculated by bring tS and tT into Eq. 11. Finally, based on the local temporal discrepancy dp and global discrepancy dg, we design a domain weight strategy to evaluate the correlations between local temporal feature discrepancy and the global temporal feature discrepancy as below.

where dgp measures the discrepancy between each local temporal feature of day p and the global temporal feature.

Overall Temporal Domain Adaptation. Final, the overall temporal feature of source city and target city can be obtained by bridging Eq. 12 and Eq. 9.

These two global temporal features focus on domaininvariant temporal features via the proposed domain weight.

Temporal Domain Loss Lt. Overall, bring ˙tS and ˙tT into Eq. 10, we define the temporal domain loss function as:

where M denotes feature dimensions.

3.4 PredictionModule

Referring to RevGrad [Pei et al., 2018], we design a prediction moduleMp to exact task-driven spatio-temporal features. In RevGrad, Mp is a classification task. In this paper, we design Mp as a FC network for crowd flow prediction.

Prediction Loss Lp. Here, we utilize the Root Mean Squared Error (RMSE) as the prediction loss function, which is widely used in various spatio-temporal prediction studies.

where Y is the ground truth and ˆ Y is the predicted result.

3.5 Overall Loss Function

Given the spatial domain loss Ls, temporal domain loss Lt, and prediction loss Lp, we bridge them together as:

where β and γ control the weight of spatial/temporal aspects.

4 Experiments

Datasets. We adopt five popular open-source urban crowd datasets, namely NYCTaxi, NYCBike, CHIBike, BJTaxi, and Chengdu, which are commonly used in related studies. Table 1 summarizes the statistics of the datasets. To validate the genericity of STAN for spatio-temporal knowledge transfer, we conduct experiments considering both intra-city and intercity cases. Specifically, we select NYCTaxi as the source while NYCBike as target; CHIBike as the source while NYCBike as target, NYCBike as the source while CHIBike as target, and BJTaxi as the source while Chengdu as target. For the source city, we use 9 months of data (NYCTaxi, NYCBike, and CHIBike) and 3 months of data (BJTaxi) for training and the rest for testing. For the target city, we use 1 month of data (NYCBike and CHIBike) and 10 days of data (Chengdu) for training and the rest for testing. Note that we use NYCTaxi → NYCBike as default due to limited space.

Parameter Settings. First, we split the whole city into region/grid maps and cut the time dimension into nonoverlapping time intervals (see Tab. 1). Then, we select 3 days of historical data where each day contains 9 time intervals. Further, we compute inflow/outflow of each region based on Eq. 1 and then normalize the flow data into [0, 1]. In terms of SAAM, the convolution kernel size is set to 3 × 3, and the domain discriminator contains two fully-connected layers with sizes are 64 and 1. In terms of TAAM, the length of LSTM and hidden feature size are set to 9 and 128. In terms of PM, two connected layers' sizes are 256 and 512. Then, we set β and γ as 0.1. As for model training, the epoch size, dropout, and learning rate are set to 32, 0.5, and 1×e-6 . Besides, after 50 epochs, we reduce the learning rate to 0.9 times the original after every five epochs. STAN is implemented with Pytorch framework on GTX-3090 24G GPU.

Comparison Baselines. We compare STAN with: i) Deep learning methods. We choose ConvLSTM [Xingjian et al., 2015] and STResNet [Zhang et al., 2017] as they are remarkable deep crowd flow prediction models. Here, we only use target city data for model training. ii) Fine-tuned methods. These methods first pre-train the deep models on the source city and then fine-tune models on the target city. We select ConvLSTM, STResNet, and DCRNN [Li et al., 2018b] and denote them as ConvLSTM-FT, STResNet-FT, and DCRNNFT. iii) Transfer learning methods. MetaST [Yao et al., 2019a] and ST-DAAN [Wang et al., 2021] achieve state-ofthe-art performance for across-city urban flow predictions. We adopt RMSE and MAE as Evaluation Metrics.

4.1 Effectiveness Study

As shown in Tab. 2, we compare different methods for both intra-city (i.e., NYC → NYC) and inter-city (i.e., CHI → NYC, NYC → CHI, and BJ → Chengdu) transfer cases. As expected, deep learning methods perform worst as the target city has scarce samples to guide model learning. Although the fine-tuned baselines perform better than non-transfer deep learning models, they are still inferior than transfer-learning methods. In addition, STAN consistently outperforms all baselines. For example, STAN achieves the reduction of RMSE by up to 15%, which verifies the benefits of our dynamic spatio-temporal modeling over static spatial-only correlation modeling of compared baselines.

Ablation Study. To evaluate SAAM and TAAM, we proceed to compare STAN with STAN-No-SAAM and STANNo-TAAM of STAN that remove SAAM and TAAM, respectively. It shows that RMSE and MAE slightly increase but still lower than baselines in both experiments, which proves the effectiveness of designed SAAM and TAAM.

4.2 Robustness Study

Effect of DataAmount. Here, we compare STANwith STDAAN, as the latter shows state-of-the-art performance. The results are shown in Fig. 3. First, STAN outperforms STDAAN under varying data samples of source and target cities. Second, more source/target data lead to better performance as more helpful knowledge can be transferred. Third, when the training data of the source/target city is smaller, the improvement of STAN is usually more significant than competitors, showing that STAN is more robust to data-scarce cases.

Sensitivity to P and k. As shown in Fig. 4, compared with ST-DAAN, STAN performs more stably with different P and k settings, further proving the superior robustness of STAN.

Tuning β and γ. Finally, we show how to set weight β and γ in Eq. 16. We set β (resp. γ) as fixed, and we change γ (resp. β) from 0.1 to 0.9. As shown in Fig. 5, reasonable settings for β and γ are 0.1. In addition, STAN always performs better than baselines with varying weight preference.

4.3 Efficiency Study

Last but not the least, we study the model efficiency. In terms of the feature embedding, as depicted in Fig. 6 (a), we replace the SAAM of STAN with MMD to embed the spatial features at each time step, resulting in STAN-MMD. As observed, STAN outperforms STAN-MMD by a large margin on memory occupy, as STAN adopts efficient adversarial classification to avoid massive feature computations.

In terms of the model training, the loss curve of the STAN training process is shown in Fig. 6 (b). One can see that the model converges quickly. Specifically, within around 20 epochs (corresponding to 10 minutes), the training loss first drops quickly and then becomes stable. This running time efficiency is acceptable in real-life deployments.

5 RelatedWork

Deep learning has become the most popular method for prediction studies [Guo et al., 2021; Yuan et al., 2021]. In terms of spatial modeling, CNN [Zhang et al., 2016] based models split a city into equal-size regions. Then, convolution kernels can be utilized to capture the spatial correlations across regions. Recently, GCN-based models [Ye et al., 2020; Bai et al., 2020; Fang et al., 2021] are proposed. GCN constructs a graph contains nodes and edges where a node may represent a road intersection and the edge represents the relations. Obviously, GCN-based models focus on road network modeling rather than urban modeling for citywide flow prediction. In terms of temporal modeling, RNNs [Xingjian et al., 2015; Ye et al., 2019] are widely applied. [Yuan and Li, 2021; Jiang et al., 2021; Ye et al., 2020] provide surveys on deep urban flow prediction. However, the deep learning approaches require massive training data and hard to generalize to support urban prediction with data-scare problems.

Transfer Learning. [Wang et al., 2019] propose the first cross-city transfer learning for traffic prediction by matching similar regions between the source and target cities. [Yao et al., 2019a] learn the period information from multiple cities for spatio-temporal learning. Obviously, they perform knowledge and prediction learning separately. To this end, [Wang et al., 2021] perform urban knowledge transfer and predictive learning jointly, which also gains state-of-the-art performance. However, they all build the models via static and spatial-only feature transfer, while ignoring the fact inter-city correlations change dynamically across time.

Acknowledgments

This work was supported by the NSFC under Grants No. (62025206, 61972338, and 62102351), and also supported by AZFT. Yunjun Gao is the corresponding author of the work.