Fairness and Representation in Satellite-Based Poverty Maps: Evidence of Urban-Rural Disparities and Their Impacts on Downstream Policy

1 Introduction

Satellite-based poverty maps are increasingly being used to inform critical policy decisions, including estimating interim subnational statistics [Hofer et al., 2020], targeting humanitarian aid [Aiken et al., 2022; Smythe and Blumenstock, 2022], determining eligibility for social services [Gentilini et al., 2022], and estimating the impacts of development programs [Huang et al., 2021; Ratledge et al., 2022]. These maps are constructed by applying machine learning (ML) algorithms to high-resolution imagery, based on the premise that the algorithm can learn to predict poverty from pixel data [Jean et al., 2016; Yeh et al., 2020; Chi et al., 2022].

However, satellite-based poverty maps are not perfect. When poverty predictions exhibit systematic errors, their use in policy decisions can lead to disparate and unfair outcomes. For example, a program that provides resources to the regions of a country with lowest predicted wealth might disproportionately "miss" poor regions with substantial infrastructure and large, developed settlements signaling wealth from the sky. In such cases, the use of current satellite-based poverty maps - which in principle could be used to address the United Nation's (UN) Sustainable Development Goals and other pressing social issues - might in practice conflict with goals of promoting equity (for example, as formalized in the UN's Leave No One Behind Principle).

The potential for satellite-based poverty maps to aid public policy thus exists alongside the potential for such predictionbased policies to introduce or exacerbate inequities. In settings where policymakers may mis-perceive satellite-based maps as technocratic and therefore "objective" measures of poverty, it is imperative to document how systematic errors and biases might arise or compound in satellite-based poverty predictions and their uses in downstream policies.

This paper explores the interconnected phenomena of systematic prediction errors, representation, and unfairness in satellite-based poverty maps, focusing on disparities between urban and rural areas: are satellite-based maps as useful for distinguishing poverty levels within urban and rural areas as between them? Do satellite-based poverty maps tend to overestimate wealth in urban areas relative to rural ones (or vice versa) - and if so, what are the consequences for downstream policy decisions based on such maps? We focus our analyses on urban-rural disparities because (1) previous work has established urban build-up as as a key predictor of poverty in satellite-based machine learning models [Yeh et al., 2020; Engstrom et al., 2022] and (2) many sensitive or protected characteristics - including race, age, and religion - are correlated with urbanization [Ghosh and Roy, 1997; Kuper, 2013].

Using survey data and satellite imagery from ten countries (Table S1), our analysis produces four main results:

First, we document performance disparities across rural and urban regions and connect them to potential representational limitations of current methods. It appears that in many countries, satellite image representations can be used to somewhat accurately differentiate between wealthy and poor regions mainly because these representations capture differences between urban areas (which tend to be wealthy) and rural areas (which tend to be poorer). As a result, satellite-based poverty maps are not as effective at differentiating wealth within rural and urban parts a country as they are at estimating wealth at a national scale.

Second, we document nuanced but systematic biases in prediction errors for urban and rural areas. In countries where poverty is concentrated in rural areas, predicted wealth in urban areas is under-ranked relative to predicted wealth in rural areas. In contrast, in countries with a high degree of urban poverty, predicted wealth in urban areas is consistently overranked relative to predicted wealth in rural areas.

Third, we study how these phenomena interact to impact the fairness and effectiveness of downstream policies based on predicted maps. We simulate hypothetical geographically targeted aid programs which select beneficiary regions using satellite-based poverty predictions. We observe two contrasting phenomena with opposite effects on selection policies, both tied to the underlying joint distribution of urbanization and wealth. First, systematic over-ranking of rural wealth results in under-allocation of aid to rural areas (particularly when there is a strong correlation between urbanization and ground-truth wealth). Second, overreliance on weaker correlations between urbanization and wealth (arising from representational limitations in satellite imagery) may result in "missing" some of the urban poor.

Fourth, and finally, we explore options to reduce the exposed disparities in satellite-based poverty mapping. We find that simple recalibration methods can improve predictive accuracy and ameliorate prediction biases in some contexts, but rely heavily on having reliable measures of regions being urban or rural with which to recalibrate.

1.1 RelatedWork

Satellite-based poverty maps - which have been studied in the research literature for some time [Jean et al., 2016; Yeh et al., 2020; Chi et al., 2022; Rolf et al., 2021] - are now being used in real-world policy decisions, including the geographic targeting of social assistance (in Togo [Aiken et al., 2022], the Democratic Republic of the Congo [Gentilini et al., 2022], and Malawi [Paul et al., 2021]) and policy impact evaluation (in Uganda [Huang et al., 2021] and Rwanda [Ratledge et al., 2022]). Broad calls to consider fairness and responsibility in satellite-based machine learning - e.g. in environmental applications [McGovern et al., 2022], big data for development [Blumenstock, 2018], and remote sensing [Burke et al., 2021] - underscore the importance of evaluating fairness and potential biases in these maps.

While the implications of algorithmic biases have been documented in settings from criminal justice [Chouldechova and G'Sell, 2017] and facial recognition [Buolamwini and Gebru, 2018] to credit scoring [Liu et al., 2018] and resource allocation in healthcare [Obermeyer et al., 2019], they have received relatively little attention in the domain of poverty mapping. Recent studies have highlighted specific fairness concerns for particular regions and applications: Kondmann et al. [2021] investigate statistical bias in estimation of poverty and electrification rates across villages in rural India, Zhang et al. [2022] expose performance gaps of unsupervised transfer learning for landcover classification across rural and urban regions of China, and Smythe and Blumenstock [2022] evaluate satellite-based poverty targeting in Nigeria.

However, to date there exists no systematic study of broader fairness concerns in satellite-based poverty mapping - partly because the data context of low- and middle-income countries (LMICs), where the utility of satellite-derived maps is most distinct, makes it difficult to rigorously evaluate map accuracy and fairness [Jerven, 2013; Bolliger et al., 2017; Burke et al., 2021; Rolf, 2023]. Our work builds on previous studies by concretely illustrating how errors and biases in satellite-based poverty maps can translate into disparate outcomes for downstream policy decisions.

2 Data andMethods

Our analysis relies on survey datasets from ten countries matched to featurizations of satellite images.

2.1 Survey Datasets

We use survey datasets from ten countries in our paper, described in detail in Appendix A and Table S1. In short, we use the following four categories of survey data:

Demographic and Health Surveys (DHS) from Colombia, Honduras, Indonesia, Nigeria, Kenya, the Philippines, and Peru. Each survey was conducted in 2010 or later and interviewed 20,000-60,000 households in 1,000-5,000 clusters. Clusters are small geographic groups of households, sampled at random or stratified random in each country. Clusters are roughly equivalent to a neighborhood in urban areas (for which the provided cluster centroid is jittered with a 2km radius) or a village in rural areas (for which the jitter is a 5km radius). We use the DHS-constructed asset-based wealth index as the ground truth measure of poverty for each DHS survey, and calculate the average wealth index for each cluster.

The American Community Survey (ACS) from 2018, which interviewed 1.5 million randomly selected households from all 2,331 Public Use Microdata Areas ("PUMAs") in the United States. We use household income as the ground truth poverty measure in the ACS, and calculate the average household income per PUMA.

The Mexican Intercensal Survey from 2015, which interviewed 2.8 million households in Mexico's 2,446 municipalities. We construct an asset-based wealth index from the survey data, using a principle components analysis to project ownership of twelve assets to a unidimensional vector (Appendix A.1). Our ground truth measure of poverty in Mexico is the average asset-based wealth per municipality.

The Indian Socio Economic and Caste Census (SECC) from 2012. We use estimated average per-capita consumption at roughly the village/town level (shrid2) produced by the Socioeconomic High-resolution Rural-Urban Geographic Dataset for India (SHRUG) v2 (an updated version of [Asher et al., 2021]) as our reference measure of poverty. We spatially aggregate small rural shrid2 regions together (Appendix A.2) to ensure each observation is a large enough geography and to reduce imbalance between the number of urban and rural regions. This reduces the number of rural observations from 522,344 to 59,832. There are 3,524 urban regions.

We normalize the poverty values for each country (logged in the US and India 1 ) to zero mean and unit variance. We refer to these poverty measures as "wealth" throughout.

Categorizations of each region as either urban or rural are defined by these survey datasets. We refer to these binary labels as "urbanization" throughout.

1 We log poverty values in the US and India as these values represent consumption distributions, which are right-tailed. In the remaining countries, poverty is measured with asset indices and we do not use a log transform.

2.2 Satellite Image Features

We obtain a set of tabularized features summarizing satellite tiles in each country we study from MOSAIKS [Rolf et al., 2021], accessed via siml.berkeley.edu [Carleton et al., 2022]. The underlying satellite images are from Planet Labs in 2019. 2 Features are generated through an unsupervised machine learning approach based on random convolutional features (RCFs), which are shown to carry skill across a variety of prediction tasks [Rolf et al., 2021].

RCF embedding functions are essentially a wide and shallow feed-forward convolutional neural network with random but fixed (non-optimized) weights. We use RCFs as convenient way to obtain images features with a single, fixed featurization method across countries.

The number of tiles per region varies widely between survey datasets: in the DHS, where each cluster has a 2-5km radius, each cluster is represented with 16-88 tiles. In the India, Mexico, and the United States, regions can overlap as few as six tiles or as many as tens of thousands of tiles (Table S1). For regions that intersect more than 100 tiles, we take a random subset of 100 of the intersecting tiles. We then calculate the average of each MOSAIKS feature for each region, weighted by the overlap between the tiles and the region.

2.3 ProblemFormulation and Simulation Setup

Our machine learning simulations begin by randomly assigning 75% of regions in each country to a training set and 25% to a test set. 3 Following Rolf et al. [2021], in each country we train a ridge regression model to predict average household wealth in training set regions from the associated satellitederived MOSAIKS features. The objective function is mean squared error, and we tune the ℓ2 penalty via three-fold crossvalidation on the training set. We then use the trained model to predict wealth for every region in the test set. To account for idiosyncrasies in random train-test splits, we report the mean ± two std. errors across 100 simulations in all results. 4

2.4 Fairness Analysis Procedures

Our analysis focuses on bias and fairness in satellite-based poverty maps along urban-rural lines. First, we document performance disparities within and between urban and rural areas, by measuring predictive accuracy (measured with R2 and Spearman's ρ) in the test set overall, in just urban regions, and in just rural regions. Second, we measure systematic prediction biases between urban and rural regions when using satellite-based poverty maps, quantified as (1) the mean signed error in wealth prediction for rural and urban areas separately, and (2) the mean error in wealth ranking for rural and urban areas separately.

2 Satellite imagery (from 2019) is not obtained from the same time period as all survey datasets, which range from 2010 to 2019. Other work suggests that the impacts of this temporal mismatch are limited [Yeh et al., 2020], and we observe no clear relationship between predictive accuracy and temporal mismatch in Figure 1.

3 We use uniform random assignment of regions (PUMAs in the US, municipalities in Mexico, aggregated Shrid2 units in India, and clusters in DHS surveys) to train and test sets - rather than spatial stratification - as it allows for more consistency across countries, and better reflects the "in-sample" scenarios in which satellite-based poverty maps would be deployed [Wadoux et al., 2021; Rolf, 2023].

4 Replication code is available at https://github.com/emilylaiken/ satellite-fairness-replication.

We then measure how performance disparities and prediction biases propagate to downstream policy decisions. We simulate hypothetical aid programs using satellite-based poverty predictions to select eligible geographies. To evaluate the implications of performance disparities on simple metrics of allocational fairness, we compare the precision and recall (equal by definition in this application [Brown et al., 2018]) of hypothetical programs that target the poorest 20% of regions in each country as a whole, the poorest 20% of urban regions, and the poorest 20% of rural regions. To show how systematic prediction biases propagate to downstream policy decisions in nationwide aid programs, we measure aid allocation (measured as the number of regions selected) to rural areas and urban areas when satellite-based poverty maps are used to select geographies, and compare to allocations when ground truth measures of poverty are used.

2.5 Recalibration Approaches

We explore two recalibration-based options for addressing fairness issues in satellite-based poverty prediction: mean calibration (adjusting the means of urban and rural predicted wealth distributions to match the means of the ground truth distributions), and selection threshold calibration (allocating resources to urban and rural areas according to the share of regions that are poor in each group). For both approaches, we learn the parameters of the calibration procedure on the training set, and apply this learned calibration to the test. We investigate whether access to ground-truth urbanization values affects the results our calibration approaches by also attempting calibration with predicted urbanization in test regions.

3 Results

3.1 Performance Disparities and Representation

Consistent with past work [Yeh et al., 2020; Engstrom et al., 2022], we find that satellite-based wealth predictions explain a significant portion of the variance in ground-truth wealth within each of the ten countries we study (mean R2 = 0.470.70), and there is a strong correlation between wealth predictions and ground truth (mean Spearman's ρ = 0.71-0.83).

In all ten countries, the rank correlation is substantially lower when predictions are evaluated just within urban areas (mean ρ = 0.51-0.74) or just within rural areas (mean ρ = 0.40-0.82) (or both, Figure 1A).This systematically replicates analysis in [Yeh et al., 2020] (which documents performance within-urban and within-rural areas for a pooled dataset from several African countries) for ten countries across the globe. There is heterogeneity across countries in terms of which areas are hardest to predict: in three countries (Colombia, Peru, and the United States) predictive accuracy is higher among urban areas than among rural areas, whereas in the remaining seven countries (Honduras, India, Indonesia, Kenya, Mexico, Nigeria, and the Philippines) predictive accuracy is higher among rural areas. In all countries, at least one of urban or rural areas has substantially lower predictive accuracy than the country as a whole (difference in mean ρ > 0.09, Figure 1A).

Why are satellite-based poverty maps consistently worse at differentiating poverty levels within urban or rural areas than within entire countries? Trends in the imagery and observed wealth data point to the possibility that much of the accuracy observed in country-scale satellite-based poverty maps is due to their ability to distinguish between urban and rural areas.

In each country, there is a strong correlation between the measured ("ground truth") values of wealth and urbanization (Table S3, Spearman's ρ = 0.51-0.77 outside of India and the United States). 5 We also find that the overall performance of poverty predictions tends to be higher for countries where wealth and urbanization are more correlated (Figure S5). 6

5 In the India and United States, ρ = 0.28-0.30. The United States is the only high-income country of the ten we study. The relatively low correlation between wealth and urbanization in India in our data might be due in part to the definition of shrid2 regions, in which many urban regions have large spatial extent while a large majority of region instances are rural (see Appendix A.1).

6 This trend does not hold, and possibly reverses, when evaluating across only urban or rural regions (also Figure S5).

The potential influence of urbanization can also be seen in the feature representations of the raw imagery - even before fitting a predictive model -which already encode high amount of signal as to whether a region is urban or rural (Figure 2B). As shown in Figure 2A, the average ℓ2 distance between features of two rural regions is much lower than that between an urban and a rural region (and two urban regions). We find that a similar overall trend holds when looking at individual MOSAIKS tiles (Figure S1), and that satellite imagery is highly predictive of urbanization (Table S3).

Finally, in countries where wealth and urbanizaton have a strong correlation, the differences between the predictive accuracy of satellite-based wealth predictions and satellitebased predictions of a region being urban are small (mean difference in Spearman's ρ = 0.07-0.26 outside of the United States and India, Table S3 and Figure S3). Along with the results in Figure 2, the close relationship between predicting urbanization and predicting wealth from satellite imagery hints at potential concerns about representations of poverty in satellite imagery akin to stereotype bias [Abbasi et al., 2019;

Boyarskaya et al., 2020], a particular type of representational harm in which the observed data on individuals in a group are more closely related than a more comprehensive characterization of those individuals would warrant.

Taken together, these results suggest that representations of poverty in satellite imagery beyond urbanization are present but often limited. As such, a concern for policy is that applications that "zoom in" on urban or rural areas (for example, calculating interim subregional poverty statistics or running an aid program in just urban or rural areas), predictive accuracy for identifying poverty from satellite imagery - and the accuracy of downstream decisions - is likely to be substantially lower than an overall accuracy estimate would suggest.

3.2 Systematic Biases in Prediction Errors

In light of the limitations to poverty representations in satellite imagery, a further concern for satellite-based poverty mapping is possible systematic biases in prediction errors.

We begin by documenting mean signed errors in predictions, finding that across countries, wealth in urban areas is under-predicted and wealth in rural areas is over-predicted (Figure 3A, Figure S4). This phenomenon may simply reflect a statistical bias toward the mean prediction - in all countries urban areas are on average richer than rural areas (Table S3).

The mean error in wealth ranking across countries exhibits biased errors in both directions: in Nigeria, the Philippines, and the United States, rural areas are under-ranked by wealth predictions; in Colombia, India, Kenya, Mexico, and Peru, rural areas are over-ranked; and in Honduras and Indonesia, there is no statistically significant difference in ranking between urban and rural areas (Figure 3B).

An important question is whether these same biases could arise if simply using a lower-quality wealth label, rather than satellite-based predictions. Figures 3 and S4 therefore include noised-wealth baselines, in which we add Gaussian noise to the ground-truth wealth labels with zero mean and isotropic covariance calibrated to the mean squared error of the satellite-based predictions. This allows us to test whether prediction biases of satellite-based models are systematically different than those that would be observed under a model of independent, additive prediction noise. Since urban areas have higher average wealth than rural areas across countries in our study, we expect the noised income baseline will overrank rural wealth and under-rank urban wealth.

Both the satellite-based poverty predictions and the noisedincome baseline over-rank wealth in rural areas in most countries (horizontal axis of Figure 3B). The degree of overranking tends to be higher for the noised baseline than the satellite-based predictions. The notable exceptions are the United States and the Philippines, where prediction biases from satellite imagery run in the opposite direction of those from the noised wealth baseline (wealth is under-ranked in rural areas by satellite-based predictions and consistently overranked by the noised wealth baseline in these two countries). We explore possible drivers of these differences in Section 4.

3.3 Implications for DownstreamPolicies

To study the extent to which performance disparities and systematic prediction biases can propagate to allocative unfairness in downstream policy decisions, we simulate hypothetical geographically targeted aid programs in each country, as described in Section 2.4.

Geographic targeting effectiveness. We find that the disparities in predictive performance between urban and rural areas documented in Section 3.1 reduce the effectiveness of downstream decisions made using the satellite-based poverty predictions. A simulated social protection program aiming to select the poorest 20% of regions nationwide using satellitebased poverty maps tends to have relatively high recall and precision (54-71%), whereas programs identifying the poorest 20% of regions within urban or rural areas have lower recall and precision (38-73% in rural areas and 46-65% in urban areas, Figure 1B).

Allocative unfairness. The systematic biases in ranking of poverty by satellite-based predictions (Section 3.2) suggests a risk of allocative unfairness when using satellite-based poverty predictions to inform policy. In our simulated nationwide aid programs, in countries where the relationship between urbanization and wealth is strong (Colombia, Honduras, India, Indonesia, Kenya, Mexico, Nigeria, and Peru), aid tends to be under-allocated to rural areas (by 1-5 percentage points) compared to what would be allocated using ground truth wealth from the survey data. In countries where correlation between urbanization and wealth is weaker

(the Philippines and the United States), aid tends to be overallocated to rural areas (by 2-3 percentage points, Figure 3 and S4). This latter pattern runs in the opposite direction for the noised wealth baseline (faded markers in Figure 3), indicating that error structures specific to satellite-based wealth predictions are driving allocative unfairness, rather than general degradation of wealth estimates.

4 Drivers of Allocative Unfairness

The nuanced patterns of allocative unfairness in Section 3.3 can be at least partially explained by characterizing two phenomena driving errors in satellite-based predictions and ranking of wealth between urban and rural areas:

Reversion towards the (sample) mean. One possible driver of allocative unfairness is that predicted wealth can be biased upward for low wealth regions and downwards for high wealth regions, towards the overall mean wealth value in the training data (as described in Section 3.2). In our simulated aid program, the upward bias of wealth rankings in rural areas results in under-allocation of aid to rural areas. Colombia, Honduras, India, Indonesia, Mexico, Nigeria, and Peru are all emblematic of this pattern to varying degrees. Notably, allocative biases are less severe for many of these countries with satellite-based errors than would be expected with classical Gaussian prediction errors (simulated with the noised wealth baseline in Figure 3). One possible explanation for this pattern is that a second driver of allocative unfairness in satellite-based poverty predictions - described below - works in the opposite direction of classical prediction error.

Reliance on correlations between urbanization and wealth. A second potential driver of allocative unfairness is a limited predictive power beyond identifying built-up areas (established in Section 3.1). If variation in predicted wealth is driven by urbanization, whereas variation in true wealth is driven by more factors, satellite-based poverty prediction algorithms might "miss" populations of urban poor, having associated them with urbanized regions tending to be wealthy. The United States and the Philippines - which have the lowest and third-lowest correlation between urbanization and wealth of all the countries we study, and the lowest overall prediction performance (Table S3) - demonstrate this pattern.

While these two phenomena have different effects on the allocation rate to urban and rural areas, it is possible (and likely) for them to manifest jointly. 7 Summarized in Figure 3, for most countries the first driver seems to have the dominant effect on allocation rates, excluding the United States and the Philippines, where the allocative differences appear to be driven mostly by the second phenomenon.

5 Addressing Allocative Unfairness

We test two approaches to addressing the issues of allocative unfairness characterized in Section 3.3.

First, when we know which regions are classified as urban or rural, we can recalibrate the prediction distributions within urban and rural areas to align with the true per-group means in the training data. This addresses the "reversion to the mean" phenomenon in an application-agnostic way. We refer to this procedure as mean recalibration, and implement it by learning an additive offset for each group so that the predicted mean in each group matches the true group mean.

A second option is to directly address allocational unfairness in the context of resource allocation by setting different eligibility thresholds for urban and rural regions. We refer to this option as selection threshold calibration, and implement it by setting per-group allocation thresholds to match the fraction of allocations that would be sent to urban and rural areas using the reference wealth label values of the training set.

Mean calibration. Figure 4 shows that applyingmean calibration often produces downstream allocations that are closer to allocations based on ground-truth wealth measures. Mean calibration successfully reduces systematic prediction bias across urban and rural areas, and even slightly increases population level performance for some countries (increase in R2 of 0.00 - 0.02, increase in Spearman ρ of 0.00-0.02; Figure S6).

7 We discuss this issue further and propose summary statistics to help measure causes of each driver in Appendix B.

However, there are two important caveats to the mean calibration strategy. First, it only addresses the first driver of unfairness in Section 4 - reversion towards the mean. Across countries, mean recalibration increases the allocation to rural regions (evidenced by points above the y = x line in Figure 4) due to the increased separation between predicted wealth of rural and urban regions. In countries where the dominant trend affecting allocation rates is missing the urban poor (the Philippines and the United States), deploying this recalibration strategy can exacerbate allocative differences. For simulations in Mexico, mean recalibration also introduces an allocative bias toward over-targeting rural regions that was not present in the original uncalibrated predictions.

Second, this simple mean recalibration strategy works only when ground truth labels for being urban or rural are known everywhere (that is, everywhere that the satellitebased poverty map will be used - not just in the training set). When we use satellite-based predictions for whether a region is urban or rural to perform mean recalibration in the test set, allocative bias is not significantly improved in most countries (non-filled-in points in Figure 4A).

Selection threshold calibration. When using ground truth indicators of urbanization, threshold calibration results in allocations that are close to what would be allocated with knowledge of true wealth values (confidence intervals for filled-in points in Figure 4B all overlap the y = 0 line). This should be expected in our experimental setup, so long as the distributions of urban and rural wealth in the training set match those in the test set.

When satellite-based predictions for urbanization are used to perform selection threshold calibration in the test set, allocative bias is not improved - the same pattern observed in mean recalibration. It is possible that since wealth predictions and urban build-up predictions are closely related (Section 3.1), there is little additional signal in urban build-up predictions that is useful for calibration.

6 Discussion

Our work raises and investigates two main concerns relevant to researchers and policymakers interested in building and deploying satellite-based poverty maps for policymaking.

First, there are performance disparities in predictive accuracy for identifying wealth levels within urban and rural areas in comparison to between them, explained partly by somewhat limited representations of poverty in satellite imagery beyond urbanization. In particular, wealth is better differentiated between urban and rural areas than within urban or rural parts of a country (Figure 1A). Simulated aid programs that target only urban or only rural areas have lower recall than national-scale programs that can leverage the differences in urban and rural wealth (Figure 1B).

The main implication of this result for real-world deployments is that while satellite-based poverty programming at a country scale may be relatively accurate (as documented in past work [Jean et al., 2016; Yeh et al., 2020; Chi et al., 2022]), effectiveness may be substantially lower if programs are deployed just for urban or rural areas (as is fairly common in anti-poverty programming [Lindert et al., 2020]).

For researchers in machine learning, our results suggest that a focus on building predictive models that represent and distinguish wealth levels within urban and rural areas will be essential for making satellite-based poverty maps a useful and fair measurement tool. Other digital data sources, such as mobile phone data [Blumenstock et al., 2015; Steele et al., 2017], social media data [Fatehkia et al., 2020; Chi et al., 2022], or information from crowdsourced maps [Tingzon et al., 2019] may be helpful for improving representation and within-urban and within-rural differentiation.

Our second main finding is that systematic prediction biases in poverty predictions between urban and rural areas can result in allocative bias in downstream policy decisions. The direction of prediction biases and downstream disparities in allocations depends on the underlying joint distribution of poverty and urbanization: satellite-based poverty maps may "miss" populations of urban poor in countries with pockets of urban poverty, whereas in countries where poverty is concentrated in rural areas, policies based on satellitebased poverty maps are likely to over-allocate aid to urban areas. The main implication of this result for policymakers is that urban-rural biases may be present even in national-scale policies using satellite-based poverty maps, and such maps should always be audited for bias before deployment.

We test two simple yet promising approaches to addressing systematic prediction biases through recalibrating predictions or selection thresholds, but both rely on having access to ground-truth labels for regions being urban or rural in all areas where the map is deployed. Imputed urban/rural values are available at an increasingly high resolution globally [Rao and Molina, 2015]; evaluating whether such estimates are sufficient for model recalibration will be an important topic for future work. More generally, more sophisticated statistical approaches to addressing prediction bias may improve upon the ones we propose here [Proctor et al., 2023].

The real-world implications of performance disparities and prediction biases for downstream analyses and policies are likely to be multi-faceted. We study in detail the implications for one downstream use of satellite-based poverty maps: the geographic targeting of humanitarian aid. A similar analysis could be applied to understand implications of disparities and biases for other uses of satellite-based predictions, such as the estimation of sub-national statistics [Hofer et al., 2020; Sherman et al., 2023] and causal inference on the effects of anti-poverty programs [Huang et al., 2021; Ratledge et al., 2022].

In summary, we find consistent evidence of disparities in satellite-based poverty maps across ten countries, with different social structures, time scales, and modes of ground truth data collection. An important complementary analysis, however, would seek to understand how the disparities we identify interact within a single complex sociopolitical context. For example, we studied disparities only across urban and rural areas; developing a more comprehensive set of concerns will crucially rely on local settings of model use. Such contextdriven work, along with the empirical results presented here, can help policymakers realize the potential of satellite-based poverty mapping while mitigating the risk that such maps introduce bias or amplify existing inequities.

Ethical Statement

This paper seeks to expose and quantify a potentially critical ethical issue in satellite-based poverty prediction: issues of fairness within and between urban and rural areas. However, our work here still sits squarely within computational and algorithmic aspects of fairness. By focusing on trends across ten very different countries, the analysis in this paper is largely devoid of the full social context of poverty mapping and aid allocation in the each individual country we study. Country-specific and human-centered work on local conceptions of fairness in such policies will complement the analysis in this paper.

Acknowledgments

Aiken acknowledges support from a Microsoft Research PhD Fellowship. Blumenstock acknowledges support from the National Science Foundation under CAREER Grant IIS1942702. Rolf acknowledges support from the Harvard Data Science Initiative and the Center for Research on Computation and Society.

We thank Paul Novosad and Sam Asher for sharing with us with an early release of the SHRUG v2 dataset, and for feedback on an earlier draft of this work. We thank Gabriel Cadamuro, Tamma Carleton, Guanghua Chi, and Jonathan Proctor for helpful feedback on the paper.

Contribution Statement

Aiken and Rolf co-led the work, including conducting data cleaning and machine learning experiments, calculating accuracy and bias metrics, designing recalibration approaches, and writing the paper. Blumenstock provided feedback on analysis and paper.