MuseCL: Predicting Urban Socioeconomic Indicators via Multi-Semantic Contrastive Learning

1 Introduction

Urbanization is intricately connected to critical facets of the United Nations Sustainable Development Goals (UNSDGs), affecting energy, environment, economy, climate, etc. [Sachs et al., 2022]. By 2020, over 55% of the global population resided in urban areas, and this trend is projected to persist and intensify in the forthcoming decades [Habitat, 2022]. Embracing urbanization yields numerous advantages, including a thriving cultural milieu, enhanced job prospects, and improved transportation networks, etc. However, it also begets a host of predicaments and hurdles, such as air pollution, traffic congestion, and escalated energy consumption [Zheng et al., 2014]. To address these challenges and achieve SDGs, gaining a comprehensive understanding of the urbanization phenomenon through fine-grained region profiling and accurate socioeconomic indicators becomes crucial.

∗Corresponding author.

Traditional approaches have relied on community surveys to gather statistics on metrics like population density and household income, which is both resource-intensive and timeconsuming [Custodio et al., 2023]. With the maturation of urban perception technology, diverse forms of data continue to proliferate within cities, which paves the way for fresh opportunities in tracking urban sustainable development indicators. As depicted in Figure 1, these datasets encompass point of interest (POI) information, vehicle movement trajectories, remote sensing data, street view imagery, and social media insights, among others. The utilization of this diverse data pool offers robust support for a myriad of downstream tasks. For instance, social media data is instrumental in predicting crime and unemployment rates [Antenucci et al., 2014;

Aghababaei and Makrehchi, 2016], urban lifestyle mining [Zhou et al., 2018], and significantly contributes to studies on urban sustainability [Ilieva and McPhearson, 2018]. Trajectory data reveals valuable insights into mobility patterns, socioeconomic indicators, and health trends [Cohen et al., 2016; Gao et al., 2017; Zhou et al., 2017; Zhou et al., 2023; Wang et al., 2018b]. POI data aids in discovering new venues to explore [Zhou et al., 2019], deducing regional functions [Yuan et al., 2012], and controlling light pollution [Zhang et al., 2024]. Recent research also delves into the potential of urban imagery, employing remotely sensed images for poverty prediction, land cover classification [Jean et al., 2019; Hong et al., 2020; Burke et al., 2021], and analyzing street view images to estimate pedestrian volume [Chen et al., 2020].

However, utilizing unimodal urban data often yields suboptimal results, prompting a growing inclination towards the integration of multi-modal data. For instance, the simultaneous utilization of streetscape and remote sensing imagery has proven effective in predicting socioeconomic indicators [Wang et al., 2018a; Li et al., 2022]. The combination of urban imagery with POI data has demonstrated its utility in enhancing region representation [Wang et al., 2020; Huang et al., 2021; Liu et al., 2023]. Furthermore, researchers have ventured into the realm of multi-view graphs, leveraging data from diverse sources to comprehensively characterize regions [Qu et al., 2017; Fu et al., 2019]. This shift to multi-modal approaches holds great promise for advancing urban data analysis and interpretation, and helps to better achieve sustainable development goals. Recent efforts [Jean et al., 2019; Wu et al., 2022; Liu et al., 2023] aim to derive latent embeddings for individual regions and employ them in conjunction with regional characteristics to predict a range of socioeconomic indicators, showcasing their notable versatility.

While prior studies have undertaken region profiling and socioeconomic prediction, several challenges persist. Among these, three primary ones emerge: (1) Rapid societal development has reshaped information exchange among regions, prompting a reassessment of the applicability of Tobler's First Law of Geography [Miller, 2004]. Consequently, a more precise method is warranted to assess region similarity. (2) Urban representation predominantly focuses on geography and human activity, necessitating effective modal filtering to meet region representation demands amidst the abundance of urban data. (3) Achieving effective fusion of diverse modal data is crucial yet complex in developing the final region representation, necessitating the advancement of robust multi-modal fusion techniques.

To tackle these challenges, we present a Multi-Semantic Contrastive Learning (MuseCL) framework. The primary contributions of our work can be summarized as follows:

• We pioneer the joint representation of regions using both street view and remote sensing imagery, concurrently integrating POI and mobility flow data to enrich the embedding with multi-dimensional semantic information.
• We enhance the spatial contrastive learning process by factoring in the similarity between regional POI and population mobility, resulting in more effective contrastive learning outcomes.
• We devise a cross-modal fusion model that aligns imagery with textual representation outputs, seamlessly integrating textual semantics into imagery representations.
• We validate the effectiveness of our framework through experiments on socioeconomic indicators in three major metropolises. The results demonstrate the superior performance of our model compared to various competitive state-of-the-art baselines across multiple downstream prediction tasks.

2 RelatedWork

Urban Representation Learning. With the increasing availability of urban data, representation learning in urban areas has witnessed significant growth in recent years. Numerous studies have capitalized on the proximity of similar regions in the embedding space to address various downstream tasks, such as crime prediction [Wang and Li, 2017; Zhang et al., 2021], land cover classification [Yao et al., 2018; Luo et al., 2022], and socioeconomic feature prediction [Wang et al., 2018a; Li et al., 2022], among others. In this context, various strategies have emerged for urban region representation. For instance, Feng et al. [2017] proposed a latent representation model POI2Vec to jointly model the user preference and POI sequential transition influence for predicting potential visitors for a given POI. Wang and Li [2017] introduced a method incorporating temporal dynamics and multihop transitions. Zhang et al. [2017] presented a novel crossmodal representation learning method, CrossMap, which uncovers urban dynamics with massive geo-tagged social media data. Yao et al. [2018] proposed a framework to learn the vector representation of city zones by leveraging largescale taxi trajectories. In a similar vein, Fu et al. [2019] explored multi-view spatial networks, considering geographical distance view and human mobility connectivity view for POIs within each region. Additionally, Wang et al. [2020] devised a multi-modal and multi-stage framework integrating image and text data within the neighborhood. These diverse approaches offer unique perspectives and valuable insights for further research in the urban representation learning field.

Socioeconomic Indicators Prediction. Initially, researchers primarily employed supervised and unsupervised learning methods for predicting socioeconomic indicators. For instance, Chakraborty et al. [2016] proposed a generative model of real-world events to predict various socioeconomic indicators based on extracted events. Qu et al. [2017] introduced a multi-view representation learning approach that fostered collaboration among different views to generate robust representations, subsequently used for socioeconomic indicator prediction. Similarly, He et al. [2018] unveiled correlations between visual patterns in satellite images and commercial hotspots. In recent years, self-supervised learning methods, especially contrastive learning, have gained traction for socioeconomic indicator forecasting. Drawing inspiration from Tobler's First Law of Geography [Miller, 2004], Jean et al. [2019] employed distance to establish neighborhood similarities in loss functions. Furthermore, Xi et al. [2022] incorporated POI similarity into contrastive learning to overcome distance-based limitations.

Step 1: Obtaining imagery semantic features

Step 2: Incorporating textual semantic information

Figure 2: The overall architecture of the proposed MuseCL.

3 Preliminaries &Problem Statement

An urban area typically comprises multiple regions denoted as R = {r1, r2, · · · , rN}. These regions exhibit unique geographic and demographic characteristics, often reflected through various data sources within them. In our study, we focus on analyzing specific attributes of regions ri ∈ R (i = 1, 2, · · · ,N), investigating the following aspects:

• Remote Sensing Imagery RVi. Remote sensing imagery captures ground surface details, effectively revealing building distribution and thus providing valuable support for region representation.
• Street View Imagery SVi = {si1, si2, · · · , si|SVi|}. It offers valuable insights into the appearance of streets, buildings, and their immediate surroundings. A region often contains multiple street view images.
• POI Data Ti = {Ti1, Ti2, · · · , Ti|Ti|}. We textualize each POI as a bag of words {t1, t2, · · · , tn}, where each word is obtained from the POI's categories, ratings, reviews, and other relevant information.
• Population Mobility Mi = {min i ,mout i }. min i and mout i refer to the number of people entering and exiting the region ri over a period of time, respectively. It can reflect the socio-demographic activity of a region.

Given a collection of urban remote sensing images RV, street view images SV, POI data T , and population mobility data M, our primary objective is to derive a lowdimensional representation ϵi ∈ Rd for each region ri ∈ R(i = 1, 2, · · · ,N), where d signifies the dimension of the representation vectors. By effectively encapsulating the diverse characteristics inherent in each region, our approach aims to generate compact yet informative representations, denoted as E = {ϵ1, ϵ2, · · · , ϵN}, to enhance various downstream socioeconomic prediction tasks in urban settings.

4 Methodology

4.1 Framework Overview

Figure 2 illustrates our proposed framework for fine-grained urban region profiling to predict socioeconomic indicators. This multi-step contrastive learning model consists of three key components: extracting semantic features from the visual modality, incorporating textual semantic information, and performing downstream tasks.

To begin, we partition the visual semantic learning module into remote sensing imagery representations based on POI similarity and street view imagery representations based on population flow similarity. Contrastive learning sample pairs are curated to acquire imagery features with distinct focal points. Subsequently, we take into account the POI text information associated with each region and leverage a pretrained encoder-based model to derive the text features for every region. Then, employing a feature-level attentive fusion module, we align the combined remote sensing and street view features with the text representation vectors of each region, thereby imbuing the fused features with both visual and textual semantic insights. Lastly, we evaluate the lowdimensional representations of each region across a range of downstream tasks critical for urban sustainable development.

4.2 Visual Semantic Extraction

Street view and remote sensing imagery often contain information with different emphases. For example, street view imagery can provide characteristics of the social environment and population activity, while remote sensing imagery is more oriented towards geographic attributes and surface features [Liu et al., 2023]. Therefore, we need to get the embedding of both separately and combine them effectively.

Constructing Contrastive Samples

Recently, Xi et al. [2022] highlighted the limitations of Tobler's First Law of Geography [Miller, 2004], noting that relying solely on spatial distance to measure regional similarity is flawed. To address this, we propose a refined approach by constructing contrastive learning pairs for street and remote sensing images based on population flow and POI similarity, respectively. Street images are paired with mobility data because they reflect human movement patterns, while remote sensing images are paired with POI data because they capture the built environment and land use.

Population flow within a region can be gauged by quantifying the influx and efflux of individuals or vehicles over a specified timeframe. If we conceptualize regions as nodes and the movement of individuals or vehicles between regions as edges, the mobility of each region can be captured by tallying the entries and exits at each node. Assuming that the inflow of population to region ri during a given period is min i , and the outflow is mout i , the population mobility distance between regions ri and rj is computed by:

Then, the similarity of population mobility between the two regions can be quantified as:

We can construct positive samples characterized by higher similarity and negative samples characterized by lower similarity for each street view imagery, based on the parameter λPM i,j . As for remote sensing imagery, assuming that K different POI types are considered, we employ the Euclidean distance to quantify the POI distance between region ri and rj as follows:

!"#!"

!"#!$

Therefore, the POI similarity between region ri and rj is:

POI / Mobility Triplet Loss

CNN Shared Weight Linear Relu Linear Shared Weight Using the acquired λPM i,j and λPOI i,j , we proceed to form separate pairs of contrastive learning samples for street view and remote sensing imagery. For instance, focusing on street view imagery, we establish each anchor image AncSV i along with its corresponding positive sample PosSV i and negative sample NegSV i based on the population flow similarity λPM i,j . Subsequently, we train a convolutional neural network (CNN) denoted as FSV to map the constructed contrastive learning samples C SV = [AncSV i , PosSV i ,NegSV i ] into a low-dimensional vector space: xSV i = FSV(AncSV i ), ySV i = FSV(PosSV i ), and zSV i = FSV(NegSV i ). Similarly, we derive representation vectors for remote sensing imagery denoted as xRV i , yRV i , and zRV i , corresponding to the contrastive learning samples C RV = [AncRV i , PosRV i ,NegRV i ].

Loss Optimization

To ensure the minimization of the distance between the anchor image and the positive image, while maximizing the separation from the negative image in the representation space, we employ Triplet Loss [Schroff et al., 2015] as the loss function. The primary objective of this loss function is to bring features with similar labels into close proximity within the representation space, while simultaneously pushing features with dissimilar labels apart. For each pair of samples, we anticipate the fulfillment of the following equations:

where [·]+ is a rectifier function to keep the loss function value non-negative, and sim(·) denotes the cosine similarity. The value a is used to prevent the features of anchor samples Ancmi , positive samples Posmi and negative samples Negm i from aggregating into a small space. The whole training framework is shown in Figure 3.

4.3 Textual Semantic Incorporation

POIs hold significance as data points denoting specific landmarks on a map, often signifying distinct geographic locations such as stores, restaurants, parks, and more in cities. The textual descriptions associated with POIs can effectively capture the geographic attributes of a region. For instance, a clustering of coffee shops within a region could indicate a vibrant locale appealing to young residents, whereas an abundance of parks and green spaces might suggest a neighborhood conducive to family-oriented living.

POI Textual Semantic Extraction

In addition to the imagery features, the textual data associated with POIs plays a crucial role in region profiling. To effectively harness the descriptive potential of POI text for region representation, we employ Gensim in conjunction with SkipGram and Huffman Softmax models [Mikolov et al., 2013] for training. The Skip-Gram model, a neural network-based word vector approach, enables the learning of word vectors by predicting the contextual information of a word. Concurrently, the Huffman Softmax model, which leverages Huffman trees, enhances the neural network's output layer, refining the overall representation process.

Considering the complexity and ambiguity inherent in POI comments, we adopt a two-phase approach to extract the textual semantics. In the training phase, we utilize all POI comments and categories to train the model. However, as we transition to the representation phase, our focus narrows to utilizing solely the categorical information associated with each POI within the target regions. Assuming that for region ri ∈ R, its POI data Ti = {Ti1, Ti2, · · · , Ti|Ti|} is the category of each POI in the region, and the final mapping of the trained model from words to vectors is W. Then the final POI embedding result for each region is:

Attentive Fusion Module

We proceed to integrate the street view features eSV i , remote sensing features eRV i , and POI features ePOI i , creating a comprehensive final representation tailored for utilization in various downstream tasks.

Firstly, with the inherent importance of both imagery features eSV i and eRV i unknown, we propose the incorporation of an attentive fusion module to derive weights for each of these representations. Considering street view features eSV i and remote sensing features eRV i from region ri, we introduce learnable parameters c, M, and b to facilitate their fusion:

where eImage i is the final representation for region's imagery feature and βm i (m ∈ {SV,RV}) are weight coefficients.

Next, in order to incorporate the textual semantic information of POIs, we refer to InfoNCE loss [Oord et al., 2018] to align the features of imagery eImage i and POI texts ePOI i :

where n denotes the mini-batch size. By optimizing the aforementioned loss function, we acquire the region imagery features eImage i and effectively integrate the semantic information of POIs. Subsequently, these obtained representations for each region can be harnessed to forecast various socioeconomic indicators.

5 Experiments

5.1 Experimental Setups

Datasets

We compile real-world datasets from three major cities: Beijing (BJ), Shanghai (SH), and New York (NY). The city regions are delineated by hexagonal divisions, with a radius of 1 km for Beijing and Shanghai, and 500 meters for New York (New York is much smaller than Beijing and Shanghai). It should be noted that our model is highly adaptable to various division shapes and scales, including road networks and Census Block Groups (CBGs).

For street view imagery, we employ the Baidu Maps API 1 for Beijing and Shanghai, and the Google Maps API 2 for New York. High-resolution (3.6-meter) remote sensing images are acquired through ArcGIS for all three cities. The POI data for Beijing and Shanghai originates from Baidu Maps, while New York's data is sourced from OpenStreetMap 3 (OSM). Socioeconomic indicators, including population density from WorldPop 4 , housing data from Lianjia 5 , and crime data from NYC Open Data 6 , are also integrated.

Baseline Models

We compare our proposed model with various unimodal and multi-modal region representation algorithms, including:

• Inception v3 proposed in [Szegedy et al., 2016]. It can extract features using convolutional layers with different kernel sizes, max pooling and batch normalization.
• Resnet-18 proposed in [He et al., 2016]. It uses residual blocks to solve the degeneracy problem of deep networks. We use the Resnet-18 pre-trained in ImageNet.
• Tile2vec proposed in [Jean et al., 2019]. It is an unsupervised learning method that uses geographic distance as a criterion for constructing contrastive samples.
• Urban2vec proposed in [Wang et al., 2020]. It uses both street view images and POI data to characterize neighborhood features.
• PG-SimCL proposed in [Xi et al., 2022]. It uses remote sensing imagery for region representation based on the similarity of geographic distances and POI distributions to perform prediction tasks on socioeconomic indicators.
• Add-svrv and Fusion-svrv. They represent the summation or attentional fusion of SV and RV embedding.
• Concat. We simply concatenate the SV, RV and POI representation results as a variant of our method.

Metrics and Implementation

We adopt rooted mean squared error (RMSE) and coefficient of determination (R2 ) for evaluation. In our experiments, we use Inception v3 as SV encoder's backbone and Resnet-18 as RV encoder's backbone with a final linear layer projecting features into the 128-dimensional embedding space. We set the batch size to 32, the learning rate to 5e-4 , and use Adam optimizer. Datasets are split into 60% training, 20% validation, and 20% test sets. All socioeconomic indicators are converted into logarithmic scale.

1 https://lbsyun.baidu.com/index.php?title=viewstatic

2 https://developers.google.com/maps

3 https://www.openstreetmap.org/

4 https://hub.worldpop.org/

5 https://m.lianjia.com/bj/ershoufang/index/

6 https://opendata.cityofnewyork.us/data/

5.2 Experimental Results

Socioeconomic Indicators Prediction

We utilize the embedding of each region as input and employ a multi-layer perceptron (MLP) to predict the socioeconomic indicators. Table 1 shows the prediction outcomes for Beijing, demonstrating a significant R2 improvement of 4.81% to 20.70% over the best baselines across the five datasets.

Specifically, the Inception v3 and Resnet-18 networks pre-trained on ImageNet achieve average R2 scores of only -0.0030 and 0.0283 across datasets, failing to adequately capture the relationships between regions. While Tile2vec and PG-SimCL show some improvements over pre-trained models with an average R2 of 0.0845 and 0.1428, they both fall short in providing a comprehensive representation solely through remote sensing imagery. Then, Urban2vec, a multimodal approach, outperforms other unimodal models with an average R2 of 0.6292 exhibiting enhanced prediction results by incorporating street view images and POI data. A comparison of Add-svrv and Fusion-svrv reveals that the use of attentional fusion module is more effective than simple summation when visual semantic is utilized. Our MuseCL notably excels in all Beijing datasets with an average R2 of 0.7100, which is 13.67% higher than Concat, highlighting successful multi-semantic integration across visual and textual modalities. This also demonstrates the effectiveness of MuseCL in representing regional attributes, leading to more precise predictions of socioeconomic indicators. Furthermore, Figure 4 shows the predicted value v.s. true value on socioeconomic indicators for Beijing, indicating that our MuseCL framework shows superior prediction effect on different indicators.

Model Adaptability to Other Cities

We expand our experimental scope to include other welldeveloped cities, thereby testing the adaptability of our model. Shanghai has 746 valid representation regions, while New York has 517. In Shanghai, our experiments cover Population Density (PD), Housing Density (HD), and Number of POIs (NP) indicators. For New York, we incorporate the widely recognized Crime (CR) dataset. Consistency in comparisons is maintained by employing the same baseline models as before. The predictive outcomes are presented in Table 2. Notably, our model consistently outperforms across cities with varying sizes and geographical characteristics. Compared to the next-best baselines, we achieve an improvement in R2 ranging from 3.77% to 19.61%. This robust performance reaffirms the adaptability of our model in addressing the demands of different city types and diverse datasets.

Visualization of Region Representations

To gain deeper insights into the embedding space, we apply principal components analysis (PCA) [Shlens, 2014] to downscale the representation vectors. We then use the Kmeans algorithm to classify the regions into three clusters and depict their spatial distribution in Figure 5. This visualization reveals that regions with different levels of development occupy distinct locations in the embedded space.

Specifically, the yellow regions are situated on the outskirts of Beijing, indicating underdevelopment and low socioeconomic attributes. Their remote sensing and street view images depict agricultural landscapes with sparse populations and limited POIs. In addition, purple regions, which are moderately developed, extend across urban and suburban areas, revealing emerging villages in their imagery. These areas maintain non-built spaces but exhibit higher population densities and POI counts compared to the yellow ones. Meanwhile, green regions, in central urban zones, include Beijing's commercial hubs, showing a highly urbanized environment with strong socioeconomic indicators.

Ablation Study

We conduct ablation experiments using three datasets each from Beijing and New York City. The results, as depicted in Figure 6, indicate that the absence of certain modalities leads to a reduction in the final prediction R2 value. Notably, relying solely on POI, street view, or remote sensing images yields suboptimal outcomes. When combining street view and remote sensing images without POI information, the performance still falls short of our model's performance, although it fares better than utilizing street view or remote sensing images individually. This reinforces the notion that various modalities contribute distinct insights for predicting downstream tasks and urban region profiling.

6 Conclusion

This paper presents a novel Multi-Semantic Contrastive Learning (MuseCL) framework that skillfully amalgamates semantic insights from visual and textual information to generate embeddings for urban regions. We showcase our model's superiority in socioeconomic indicators prediction across diverse cities and through extended experiments. While our focus is on statically depicting urban regions, it is important to acknowledge their rapid evolution due to development. Therefore, incorporating time into region representation presents an interesting path for future research.

Acknowledgments

This work was supported by Natural Science Foundation of China (NSFC Grant No. 62106274) and the Fundamental Research Funds for the Central Universities, Renmin University of China (Grant No. 22XNKJ24). We also wish to acknowledge the support provided by Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education and the Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China.