CGAP: Urban Region Representation Learning with Coarsened Graph Attention Pooling

1 Introduction

In recent years, there has been a rapid growth in urban data collection, driven by urbanization and the proliferation of mobile devices [Ghahramani et al., 2020]. Cities are composed of diverse regions where residents live, work, and engage in various activities. Optimizing the representation of urban regions can unveil internal properties and correlations within cities. Effective dense representations of urban regions can facilitate the prediction of various urban computing tasks, including criminal prediction [Kounadi et al., 2020], traffic prediction [Zhang et al., 2021b], and can inform policy-making [Zhou et al., 2017; Zhou et al., 2023b], urban planning [Zhou et al., 2018; Zhou et al., 2019; Wang et al., 2018], and sustainable development [Zhang et al., 2024]. Despite the emergence of large amounts of urban data, such as Points of Interest (POIs) and population mobility, and the growing demand for region representation learning, effectively learning representations of urban regions from such vast datasets remains a significant challenge.

∗Xiao Zhou is the corresponding author.

Previous methods have endeavored to leverage the spatial characteristics of regions and represent urban areas as structured graphical data [Wang and Li, 2017a; Zhang et al., 2019; Liu et al., 2020]. Utilizing this graphical structure, early studies have applied graph representation learning techniques such as Graph Neural Networks (GNNs) to acquire embeddings for regions [Fan et al., 2021; Han et al., 2022]. For instance, Zhou et al. [2023a] employed relation-aware Graph Convolutional Networks (GCNs) to learn embeddings for both regions and relation types. Additionally, many urban region attributes exhibit temporal and spatial characteristics. GNNs equipped with spatio-temporal feature recognition have been employed for region embedding [Han et al., 2020; Xu et al., 2021; Han et al., 2021]. For instance, Liu et al. [2022] devised a spatio-temporal GNN that integrates GNNs with multi-step dependency relations to support applications in spatio-temporal prediction within urban environments.

While graph representation learning methods have shown promise in urban computing tasks, they still grapple with major limitations inherent in GNN architectures: (1) information propagation limited to edges, resulting in flat networks [Ying et al., 2018]; (2) region nodes capable of learning their feature embeddings but failing to capture information from other regions in the graph structure [Zhou et al., 2023a]. Addressing these dual challenges within vanilla GNN frameworks remains difficult for region representation learning tasks. Some approaches attempt to enhance interand intra-region interaction modeling within GNN architectures but struggle with leveraging the full multi-modal information within urban environments [Wang et al., 2020]. Despite efforts to supplement regional features with diverse data, simply applying operations on the data cannot compensate for structural deficiencies in the model. Furthermore, while existing studies have demonstrated the utility of urban global features in downstream tasks [Wang et al., 2022; Wu et al., 2022], current graph representation learning methods fail to effectively integrate such global information [Wu et al., 2022]. For instance, Wu et al. [2022] proposed a joint learning approach that leverages inter- and intra-pattern information to enhance region representation. However, the integration of global features remains a challenge for existing graph representation learning methods.

To tackle the challenges of graph representation learning in urban region embedding, our focus lies in achieving effective information propagation among local region nodes within GNN architecture. Building on insights from existing studies demonstrating the utility of urban global features in downstream tasks [Wang et al., 2022; Wu et al., 2022], we propose Graph Neural Networks with Coarsened Graph Attention Pooling (CGAP). CGAP captures both local region features and urban global features through multi-level pooling. Specifically, it integrates global and regional features to extract graphical node embeddings, aiming to capture the interaction characteristics of the entire urban area through multilayer pooling and incorporate them with regional features via a global attention mechanism. Our proposed framework comprises three sequential modules: (1) a GNN module for obtaining the original embedding of the urban region graph; (2) CGAP, which includes multi-layer pooling and local attention units to generate coarsened graphs and global features from the original graph; (3) a global attention layer for integrating global features and original graph embeddings. Additionally, we define learning objectives related to urban computing downstream tasks to optimize learning at the end of the framework. Furthermore, urban regions exhibit diverse attributes and corresponding data, necessitating the consideration of multiple factors during training. Thus, our model, leveraging CGAP, integrates these various attributes into its learning objectives.

The key contributions of our work can be summarized as follows:

• We formalize urban region embedding as a graph representation learning task, integrating diverse urban data sources to capture complex interwoven region correlations and features for more effective urban region representation learning.
• We propose CGAP, a novel method that enhances GNNs for region representation learning by capturing both intra- and inter-region interactions through multi-layer pooling, generating urban global features and improving representation effectiveness.
• Through comprehensive experiments, we validate the effectiveness of our method across various real-world urban datasets and downstream prediction scenarios, outperforming competitive baseline models and demonstrating its capability in capturing intricate urban dynamics and enhancing predictive performance.

The remainder of our paper is organized as follows. Firstly, we provide problem-related definitions and formulate the problem. Next, we delve into the details of our framework, explaining the CGAP mechanism and demonstrating its effectiveness through experiments on real-world datasets. Following this, we compare our model with state-of-the-art methods. Finally, we review related works and conclude the paper.

2 ProblemDefinition

This research aims to learn representations of urban regions by leveraging the urban graph structure and its properties. The objective is to train region embeddings that are effectively encoded from the region information, thereby benefiting multiple downstream tasks in the urban computing.

2.1 Urban Region Graph

An urban is composed of regions that are connected. Each region has its own internal natural geographic or city attributes. With the exchange of residents between regions, the relationship between regions is formed. Therefore, the city is represented as G = (V, E) where V is the vector set to represent regions and E is the edge set. The geographic neighbor information of G is represented as N = {nij}(nij ∈ {0, 1}, ∀i, j ∈ [1, ∥V∥]) which is the adjacency matrix.

2.2 Region Attributes

The region attributes are the geographic and social features of urban regions. Especially, Point-of-Interests (POIs) are the most important regional attributes that we focus on. POI such as a shop, station, or hospital is any meaningful point (except geographical meaning) on the map. We define region attributes as P = {p1, p2, ..., p∥V∥} where pi is the number of POI in region i and ∥V∥ is the number of regions.

The interaction among regions relies on the movement of residents. Given an urban region graph G, human mobility is defined as a directed and weighted graph, and its adjacency matrix is represented as M = {mij} where mij represents the number of residents moving from region vi to vj(∀vi, vj ∈ V). Generally, we utilize a combination of POIs and mobility data and define it as multi-view data, which contains abundant city information.

2.3 Urban Graph Representation Learning

Given an urban region graph G, neighbor information N, region attribute P, and region relation M, our objective is to learn the embedding of urban region graph E = {e1, e2, ..., e∥V∥} where ei ∈ Rd is the embedding result of region i(∀i ∈ [1, ∥V∥]), d is embedding dimension. This learning task can be represented as follows:

where χ = ⟨G,N, P,M⟩, each row in E is ei and Γ(·) is the function that encodes multiple information related to city into the latent space for region representation. The urban computing downstream tasks utilize the embedding results as input to analyze the city.

3 Methodology

In this section, we introduce the details of our proposed framework for the urban graph representation learning task. First, we show the overview of our framework and outline each module. Then, we discuss the CGAP mechanism which aims at integrating the information from the local regions and the global graph. Finally, we present the learning objective function and its related urban computing downstream tasks.

3.1 Framework Overview

Our model is illustrated in Figure 1. After initializing urban region information into an original graph, GNNs with CGAP mechanism utilize the original graph to extract various global features. In order to get the local and global features in a graph, the global attention layer integrates compressed feature nodes and the whole graph into an urban region graph embedding result. Based on the urban computing-related objectives, we can train our model to learn effective urban region representations.

3.2 CGAPmechanism

To aggregate the original graph into a coarsened graph and thus extract the feature nodes, we propose the GNNs with Coarsened Graph Attention Pooling (CGAP mechanism) shown in Figure 2. Its proposal aims at making up for GNN architecture's deficiencies in dealing with urban reign. With the coarsened graph and global feature node generation process, the relation among regions is deepening and local regions in urban are becoming a node that represents interaction information. Therefore, the extracted global feature node which has global information could be provided to enhance region embedding. We first describe GNNs in region graph embedding initialization. And then we present the graph aggregation process with the local attention units.

Graph Region Embedding Initialisation

In this work, we convert G = (V, E) into G = (A, F), where A ∈ {0, 1}∥V∥×∥V∥ is the adjacency matrix, and F ∈ R∥V∥×d is the node feature matrix where each node has d features. To capture useful region feature and relation, we build upon

(Encoder)

GNNs [Kipf and Welling, 2016a] which apply the hierarchical structure to pass message. Given the random initial node feature embedding F , we update the feature embedding through the following message-passing architecture.

where H(k) ∈ R∥V∥×d is the hidden node embedding from the k step in GNN, M is the message propagation function, and W ∈ Rd×d is the trainable matrix. In M function, ˜ A = A + I, ˜D = ∑j ˜ Aij,H(0) = F, and W (k-1) is the k step training weight matrix.

For simple description in pooling process, we replace the GNNs calculation with Z = GNN(A, F) which represents the operation of multiple message passing and iteration through GNNs.

Pooling Process

The pooling process is shown in Figure 2. Neighbor nodes are aggregated into a coarsened node to represent local information containing these regions. Through the multi-layer local attention modules, we can get the coarsened graphs. Finally, we can get the urban global feature node as CGAP's output. Given the initial embedding result Z, this hierarchical pooling iteration process can be described as follows:

where hl represents the middle graph embedding and θl is the parameters of layer l. Finally, we can get the global feature node embeddinghg.

Local Attention Unit

We define the assignment attention weight matrix Sl ∈ Rnl×nl+1 where nl is the account of mid-graph nodes at layer l. Sl provides a method to assign local region nodes into a coarsened node in the next layer. Given the region embedding Z l from layer l, we apply the following function to get middle matrix X ∈ Rnl+1×d :

where ∥Al∥ is the account of local attention at layer l, αl is the hyperparameter of layer l and Z 0 is the initialisation result. In the local attention unit, we mask the nodes which are not in the local regions to ensure its focus scope. In the pooling process, each unit has its corresponding local regions.

To ensure the next layer node embedding learning the local relation, we also map the adjacency matrix information and calculate it in the unit:

where Al+1 ∈ Rnl+1×nl+1 , and A0 is the original adjacency matrix.

When the local attention unit updates Z l to Z l+1 , the interaction of local graph is based on X and its adjacency matrix Al+1 . We consider the node embedding information mapped in the same latent space, and thus transform coarsened nodes by the following equation:

where Wl , bl are the layer parameters which allow the result from local attention units to map in the middle graph latent space. We obtain the local graph information and build the middle graph composed of nodes to avoid flat problem due to increasing number of GNN layers. Through multi-layer pooling, we can get the single node embedding hg as the global feature.

Global Attention Layer

To capture both regional and global features from the CGAP mechanism, we apply a global attention method that integrates information to compute the region representations. Based on the global feature hg, our framework focuses on the important region feature which has a strong association with urban.

Formally, given the result hg from CGAP module and the region initial embedding Z, we compute the global attention to enhance region representation ˆE as follows.

whereWk,Wq,Wv are learned parameters, and ∥ is the operation to concatenate vectors into a matrix.

3.3 Learning Objectives

In training our model, we integrate region embedding effectiveness improvement and multi-task learning to formulate our objective function. We acknowledge the importance of enhancing global feature addition for the learning process. Additionally, by leveraging region representation for downstream tasks, we enable the downstream effectiveness to align with the learning function. Subsequently, we introduce distinct region embedding loss and multi-task loss functions.

Region Embedding Loss

To measure the influence of global feature addition, we define the region embedding loss function Lr. Given the original graph embedding Z and the final result ˆE, Lr can be computed by the following equation:

where ˆ ei, zi is the i region vector from ˆE,Z, and ∥ · ∥2 is the euclidean distance function.

Multi-Task Loss

We utilize various urban data to train our model, selecting mobility data and Points of Interest (POIs) based on region relations and attributes to calculate the loss function. The region representation is then decoded for downstream tasks based on their respective loss functions.

For tasks involving mobility and POI prediction, we employ Linear modules to decode the region representations. Given the region representation ˆE, these modules generate the predicted mobility ˆM and region POI embedding ˆ P as follows.

Given the human mobility adjacency matrix M = {mij} and its prediction ˆM, we can compute the probability in mobility distribution as follows.

where Pr(j|i) is the probability where resident source is region i and destination is region j, ˆ Pr(j|i) is the model prediction result.

And then we make use of cross entropy loss function to evaluate the proximity of real mobility to the predicted outputs:

To retain the relation among urban regions, we adopt the similarity of region attribute, and design POI loss function based on its embedding result. Given the region attributes P = {p1, p2, ..., p∥V∥} and POI embedding ˆ P = {ˆ p1, ˆ p2, ..., ˆ p∥V∥}, we formalize the POI loss Lpoi as follows.

As the total result, the loss value L can be computed as follows.

where β is hyper parameter to assign the loss weights in L.

Discussion

Global Urban Pooling and Ours. Compared to the traditional frameworks for learning region embeddings, our method adopts local node clustering and local attention units to learn the sampling so that the model can decide which information is worth sampling. Traditional methods and GNNs adopt the global urban graph in the pooling process. We summarize the advantages of our structure which global methods do not have as follows.

• Constrained by regional nodes, our local attention units focus on information exchange within the immediate neighborhood, avoiding the over-smoothing issue associated with high-order neighborhood information in Graph Neural Networks (GNNs).
• Each unit's receptive field is confined within a cluster, allowing for the preservation of heterogeneous information pertaining to the topology and features of urban regions.

Analyses. We also analyze the theoretical complexity of the pooling process. Given the input graph with node set V and edge set E in the pooling layer, our proposed CGAP requires storage complexity of O( 1µ ∥V ∥2 ) where one cluster has µ nodes, as region matrix is sparse. DiffPooling requires O(∥V ∥2 ). And SAGPool requires O(∥V ∥ + ∥E∥), where ∥E∥ ≈ ∥V ∥2 in urban computing because of the high complexity of the urban structure. In the time complexity calculation, we set the pooling depth to k. In this condition, CGAP requires O(kµ log ∥V ∥2 µ ) whereas DiffPool requires O(k log ∥V ∥2 ). So the complexity of our model is acceptable.

4 Experiments

In this section, we describe the details of our experiments. First, we introduce the dataset and implementation information related to the experiment settings. Then, we present the baseline models and downstream tasks in experiments. Under this condition, we evaluate the performance of our model and baseline models. At last, we analyse the results of our model and current methods, and validate the effectiveness of our approach.

4.1 Datasets

We collect the real-world urban data from New York City on NYC open data 1 datasets. Specially, we focus urban regions in Manhattan, and apply taxi trips as resident mobility. The regional division is base on the community boards. The detailed description of datasets id shown in Table 1.

1 https://opendata.cityofnewyork.us

4.2 Experiment Settings

In order to compare methods fairly, we use the Adam optimizer with the same learning rate of 1e-3 in our experiments. And neural network models are trained in 2000 epochs. During the training, we set the hidden dimension of models to 128, and the dropout to 0.5. The experiments were performed on the GeForce RTX 2080 Ti with 11G memory.

To align AI methods with the UN Sustainable Development Goals, we select crime prediction, check-in prediction, and land usage classification as downstream tasks. The crime prediction task contributes to Peace, Justice, and Strong Institutions, while the check-in prediction and land usage classification tasks are associated with Sustainable Cities and Communities.

4.3 Baseline Models

We compare our model with the following baseline approaches.

• GAE [Kipf and Welling, 2016b] uses a GCN encoder and an inner product decoder to learn interpretable latent representations for undirected graphs.
• Node2Vec [Grover and Leskovec, 2016] maps the graph nodes into a low-dimensional space to maximize the likelihood of preserving network neighborhoods of nodes, and propose a biased random walk process which explores diverse neighborhoods.
• HDGE [Wang and Li, 2017b] jointly learns the representations from a traffic flow graph and a spatial graph, and uses it to measure the relationship strengths between regions.
• ZE-Mob [Yao et al., 2018] learns region embeddings from the co-occurrence in human mobility data. And the model incorporate multiple mobility data into the modeling of zone embeddings.
• MV-PN [Fu et al., 2019] constructs multi-view POI-POI networks to represent regions, and introduces spatial autocorrelations and top-k locality into region embedding.
• MVURE [Zhang et al., 2021a] adopts the intra-region and inter-region data to construct multi-view graphs, and applies joint learning module to learn region embedding.
• MGFN [Wu et al., 2022] focuses mobility patterns by human mobility, and adopts a mobility graph fusion

module and the mobility pattern joint learning module to learn the embedding.

• HREP [Zhou et al., 2023a] proposes heterogeneous region embedding (HRE) with relation-aware GCN and prompt learning for downstream tasks to address both intra-region and inter-region correlations.
• DiffPooling [Ying et al., 2018] uses our framework which replace global feature extraction with DiffPooling method.
• SAGPool [Lee et al., 2019] uses self-attention, considering both node features and graph topology.
• ASAP [Ranjan et al., 2020]learns a sparse soft cluster assignment for nodes at each layer.
• Ours(CGAP) uses our framework with CGAP mechanism.

4.4 Experimental Results

In our experiments, we selected two downstream tasks, crime prediction and check-in prediction, for the main comparison experiments. We used several standard metrics to evaluate performance: Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and R2 , which are commonly used in regression tasks. We also applied the Lasso regression model [Tibshirani, 1996] for prediction. The performance of the comparison experiment is shown in Table 2. Our framework with the CGAP mechanism outperforms all state-of-the-art methods. Additionally, we evaluated method performance in a land usage classification task, using two metrics: Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI). The results, shown in Figure 3, indicate that our framework with the CGAP mechanism also performs well in these experiments.

From the experiment results, we deduced that GNN architectures used in urban region embedding have limitations in neighbor information extraction. Encoders like Node2Vec, which perceive neighbor nodes, perform better than methods like GAE, which focuses only on the embedding nodes. Fusion methods like MVURE and HREP, which integrate local urban information such as inter-region data, improve task performance. Compared to DiffPooling where pooling is done using an aggregation method for the graph as a whole, our approach focuses on the local graph information where local attention units are created for the specific local regions. Generally, our approach balances the hierarchical pooling approach with the task characteristics of region embedding. Among all approaches, our method performs best. We not only enhance the GNN architecture with local attention units that fuse local urban information but also add a global feature to region embedding. To validate these model modules, we designed ablation experiments.

From the perspective of data usage, methods focusing on mobility data or POI data, such as ZE-mob, MV-PN, and MGFN, face limitations in performance improvement. Specifically, methods using mobility, like MGFN, achieve better results among approaches with a single data type, even surpassing the multi-view data approach, MVURE. Therefore, we conducted ablation experiments on data to validate the effectiveness of multi-view data in the ablation study.

4.5 Sensitivity Analysis

We analyze the sensitivity of our model in the task of crime prediction. In this experiment, we utilize R2 as evaluation metric. The result is shown in Figure 4. We set hyperparameter β from 0.15 to 0.45, and find when β is 0.3, the performance of our framework is better than other circumstances.

4.6 Ablation Study

To verify the effectiveness of our modules in the framework, we conduct ablation experiments about the CGAP mechanism. We focus on the effectiveness of local attention units and global features in our framework. Local attention units afford fusion functions to integrate local information, and global feature provides generalized information. Therefore, we discuss two circumstances of the framework: 1) CGAP(L) replacing local attention units with Linear modules; 2) CGAP(No-G) without global feature input for global attention layer. The result of ablation experiments related to the CGAP module is shown in Table 3.

From this table, we can observe that local attention units effectively provide a fusion method to improve model performance. Through multi-layer compression, the relation among regions is strengthened. Global feature node is introduced to model and makes region representation learn from compression graph.

In order to clarify the multi-view data function, we also conduct the ablation experiments about data used in the main comparison experiments. We divide the training datasets into two circumstances: 1) Ours(POI) including single POI data; 2) Ours(mobility) including single mobility data. The result of the ablation experiments about multi-view data is shown in Table 4.

With the addition of mobility data, the experimental performance has a huge boost. The addition of POI data is useful but limited. Combining these two region data is the best choice to train region representation learning for downstream tasks.

5 RelatedWork

5.1 Urban Region Embedding

The proliferation of mobile devices and rapid urbanization improve the development of cities. A large number of researchers [Luo et al., 2022a; Zhou et al., 2023a; Zhang et al., 2021a] pay attention to urban computing tasks and analyze the city with a machine. Since cities afford the daily lives of their inhabitants, urban computing tasks such as traffic control [Zhu et al., 2020] are closely related to the daily functions of cities.

With the functional subdivision of different areas in the city, urban region representation learning rises from various urban computing tasks. Social characteristics of the region such as POI and resident mobility affect region embedding. As a typical indicator of regional prosperity, POI is an indispensable factor for representation learning. Researches [Fu et al., 2019] propose a POI-based embedding strategy and network to leverage region properties. Besides POI, other region attributes such as street information [Luo et al., 2022b] are used in this task. Compared with POI, mobility data is widely used in the embedding task related to urban spatial graphs. As the mobility data has spatio-temporal items, this feature is assisted in representing regions. Yao et al. [2018] pursuits to find co-occurrence in human mobility and add it into the embedding process. Wu et al. [2022] introduces the graphs with spatio-temporal similarity as mobility patterns for joint learning.

5.2 Graph Representation Learning

Graph embedding aims to transform node attributes into a lower-dimensional space, generating vector representations that effectively capture node relationships [Cui et al., 2018]. Recent research has centered on Graph Neural Networks (GNNs), extensively utilized for structured graph data [Wu et al., 2020]. Notably, Graph Convolutional Networks (GCNs) [Kipf and Welling, 2016a] integrate convolutional techniques from computer vision into GNNs, significantly influencing learned representations. Various strategies have emerged to enhance GCN performance. For instance, in spatial-temporal prediction tasks, STGCN [Han et al., 2020] leverages GCNs' adaptable propagation mechanism to learn node features. Furthermore, the Graph Attention Network (GAT) [Velickovic et al., 2017] refines neighbor aggregation through attention mechanisms.

Graph pooling is a fundamental element in GNN architectures [Mesquita et al., 2020]. It reduces dimensionality and compresses the input feature map for computational efficiency. Basic pooling methods [Xu et al., 2018] aggregate node representations through flattening techniques like summing or averaging node embeddings. Advanced pooling techniques [Ying et al., 2018; Knyazev et al., 2019] refine graph representations across multiple network layers. Notably, Knyazev et al. [2019] introduces local pooling and node attention mechanisms in each layer.

6 Conclusion

In this paper, we introduce a novel mechanism, CGAP, designed for node aggregation and global feature extraction within urban region graphs. It efficiently condenses specified local region graphs into singular nodes using a local attention unit, addressing the challenge of node aggregation in urban contexts. Furthermore, to overcome the inherent flatness problem in graph neural networks, CGAP employs a hierarchical structure. This structure not only preserves the original graph information but also integrates a global feature node into the global attention layer, enhancing the model's ability to capture comprehensive urban dynamics. Specifically, CGAP leverages data on human mobility and POIs to construct detailed region attributes and relationships within the graph architecture, facilitating a deeper understanding of region characteristics. Our experiments, utilizing realworld datasets for downstream applications, demonstrate that CGAP significantly surpasses all baseline methods in performance. Aligned with the UN SDGs, we aim to broaden our framework to include more downstream tasks, delving deeper into the capabilities of CGAP to enhance urban data analyses.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant No. 62106274 and the Fundamental Research Funds for the Central Universities, Renmin University of China under Grant No. 22XNKJ24. We also wish to acknowledge the support provided by the Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative. Xiao Zhou is the corresponding author.