Geolocating Images with Crowdsourcing and Diagramming

1 Introduction

Photos, videos, and other visual evidence documents the world we live in and provides a foundation for modern investigations in journalism, law enforcement, and human rights advocacy. This imagery is increasingly distributed online through social media. Governments post photos of political events, terrorist organizations share propaganda, and everyday people use smartphones to document crimes, natural disasters, and other important events.

Because visual evidence can be so compelling, it must be treated with skepticism. Photos and videos can be edited or shared with misleading contextual information, intentionally or by accident. Image verification is the challenging process of determining if imagery is what its surrounding context claims it to be; or if not, what it actually depicts [Barot, 2014]. One of the key subtasks of image verification is geolocation, which involves mapping the precise location in the world where a photo or video was made. Geolocation allows the investigator to determine where the image was actually made, and compare that with contextual claims about its meaning and purpose.

∗This paper is an abridged version of a paper titled "Supporting Image Geolocation with Diagramming and Crowdsourcing" that won the Notable Paper Award at AAAI HCOMP 2017.

Expert geolocators draw on many skills and resources to make these determinations [Higgins, 2014; Kohler and Luther, 2017]. The process is often manual, and sometimes tedious. Experts inspect the image for clues, such as familiar landmarks, weather, architecture, and landscapes. Text and graphics, such as logos and road signs, can often be researched online to narrow down possibilities. When these clues are not definitive, expert geolocators often turn to diagramming and satellite image analysis. They first draw an aerial diagram of the ground-level image under investigation, a spatial reasoning skill requiring substantial practice. Then, they use commercial GIS services like Google Maps to systematically search the area for distinctive buildings, roads, or other structures matching their diagram. Depending on the size and density of the search area, this process can require hours or days even for experts, and may still prove fruitless. If the image cannot be geolocated, it may not be verifiable.

In this paper, we explore how crowdsourcing can support this geolocation process, with the goal of helping an expert locate an image faster and more accurately. Crowds have proven to be effective at analyzing satellite imagery, but novice crowds lack an expert's spatial reasoning skill in recognizing ground-level features from aerial imagery. We close the gap by leveraging the diagramming technique from expert practice and adapting it for novice crowds to improve their satellite image analysis.

We demonstrate the value of our approach in a large-scale experiment (n=540). We find that giving crowds a groundlevel photo results in unacceptably poor performance, but an aerial diagram significantly improves their performance to near-perfect levels. Our crowdsourcing technique can reduce a geolocation search area by half in about 10 minutes while finding the target area 98.3% of the time. We also discuss the real-world applications and next steps for this work, including new opportunities to leverage the complementary strengths of crowds, experts, and computer vision, for complex image analysis tasks like geolocation.

2 RelatedWork

2.1 Computer Vision Approaches to Geolocation

Image geolocation is a longstanding problem of interest for computer vision researchers. IM2GPS [Hays and Efros, 2008] compares features in a ground photo to a reference dataset of 6.4 million geolocated Flickr images and outputs a distribution of the most probable regions of the earth. More recently, PlaNet [Weyand et al., 2016] takes a photo as input and, using a convolutional neural network trained on 126 million geotagged photos from the web, generates a probability for 26,000 cells in a grid covering the earth. Computer vision approaches like IM2GPS and PlaNet cannot yet consistently achieve the point-level specificity typically required for verification work, but may provide excellent starting points for expert geolocators.

Other work in computer vision seeks to bridge the gap between satellite and ground level imagery. [Ghouaiel and Lef`evre, 2016] developed a technique to automatically translate ground photos into aerial perspectives, but the approach requires panoramic photos and overall translation accuracy was 54%. [Zhai et al., 2016] trained a neural network to generate ground-level panoramas from satellite imagery. Their approach shows promise, but had limited effectiveness in handling high variability features like buildings. Unlike these approaches, we bootstrap an expert diagramming technique to translate between ground and satellite images.

Combining elements from the above categories, WhereCNN [Lin et al., 2015] used cross-view pairs of ground-level and 45◦ aerial imagery to train a neural network to localize ground-level photos. Their approach narrowed the location estimate to 1% of the search area for 7-22% of query images (depending on the city). While 45◦ imagery is not yet available in many areas, these automated results provide a point of comparison to our crowdsourced results.

2.2 Crowdsourced Image Analysis

Due to the impressive capabilities of the human vision system, crowds have been used to perform a variety of visual recognition tasks [Bigham et al., 2010; Noronha et al., 2011]. Many of these applications rely on crowds to identify everyday objects, scenes, or locations that do not require specialized knowledge. However, tools like scaffolding and computer vision have been used to help novice crowds analyze less familiar content, like graphic designs [Greenberg et al., 2015] or accessibility issues [Hara et al., 2015]. A rich source of examples comes from citizen science, where novice crowds recognize and categorize diverse natural phenomena [Wiggins and Crowston, 2014].

Satellite image analysis often leverages crowdsourcing, especially for humanitarian efforts like locating missing persons or assessing damage from natural disasters [Meier, 2015]. Studies of these projects emphasize the challenges novices face in translating their own observations into abstract representations [Zacks et al., 2000]. To overcome these challenges, researchers recommend partnerships between experts and novices [Kerle and Hoffman, 2013; Bianchetti and MacEachren, 2015], an idea we explore in this paper.

2.3 Expert Image Geolocation and Diagramming

Many types of professionals perform image geolocation, including journalists, intelligence analysts, human rights advocates, and private investigators [Barot, 2014; Brandtzaeg et al., 2016]. [Kohler and Luther, 2017] conducted an interview study with geolocation experts in diverse fields, focusing on their motivations, process, and use of crowdsourcing. Interviewees emphasized the importance of drawing diagrams as a tool for converting a ground-level photo into a more effective abstraction. One expert said he would "draw a bird's eye perspective, or a satellite image perspective, of how I think it may look like from the air. So I can then compare it with satellite imagery just to get a better impression." Another expert emphasized the difficulty of this mental translation: "Perspective distortion can throw off a novice or a beginner really easily because things that you see from the air tend not to look how you would think they would from the ground." These observations align with psychological research showing that people with high spatial ability use different cognitive strategies for mental rotation tasks [Just and Carpenter, 1985].

Building on these findings, we consider how diagramming can be adapted for crowds who lack an expert geolocator's spatial thinking skills. In the following experiment, we investigate whether giving a crowd an aerial diagram or a groundlevel photo leads to better geolocation results. We hypothesize that the diagram will yield higher true positive rates because it distills the most important features, but it will also yield higher false positives because an abstraction can potentially match more areas due to lack of discriminating details.

3 Study

3.1 SystemDesign

We built a web-based system using a Python/Django framework, a PostgreSQL database, and the Google Maps API for satellite imagery and GIS functions.

The main component of the system is the crowd interface (Fig. 1). The top left of the interface shows a type of reference material, depending on the study condition. In the photo-only condition, it shows a ground-level photo. In the diagram-only condition, it shows an aerial diagram. In the both condition, there is a toggle that allowed the user to switch between the diagram and the ground photo.

The bottom left shows a small Google Map (in Map mode) of a region with a 4×4 grid of 16 equal-sized subregions overlaid in black lines. After experimenting with different-sized regions, we found that a 4×4 grid struck an effective balance between context and effort.

The right side of the interface shows a Google Map (in Satellite mode) of the region, divided by translucent white lines into the same 4×4 grid. The user is confined to one subregion at a time, but can zoom in and out and toggle Map / Satellite mode. The user clicks a green Yes/Maybe button if it looks like a potential match, or a red No button if it does not, and then clicks Next. This advances the user to the next subregion, and marks in either red or green the corresponding subregion in the small map. The system advanced through the subregions in a Creeping Line search pattern, following best practices used in search and rescue [Wollan, 2004].

3.2 Locations

We used three locations for the study. BSB showed a crowded area near the Monumental Axis in Bras´ılia, Brazil. CLT was a highway near an overpass in Charlotte, NC, USA. LAX showed an intersection with crosswalks in downtown Los Angeles, CA, USA. We selected these locations and corresponding ground photos from a set of geolocation training materials prepared by an expert. Our selection criteria included similarly moderate difficulty and geographic and visual diversity.

3.3 Diagrams

We also needed a set of aerial diagrams corresponding to the above locations to compare to the ground-level photos. We considered using existing expert-drawn diagrams, but differences across experts and locations would be difficult to control in our experimental setting. Instead, we designed a set of diagram-drawing guidelines, informed by expert practice [Kohler and Luther, 2017; Higgins, 2014] and relevant standards [Painho et al., 2010; Kolbe et al., 2005], and used these guidelines to draw a low-, medium-, and high-detail diagram for all three locations using Adobe Photoshop. Lowdetail diagrams showed streets, roadways, and pathways. Medium-detail diagrams showed road markings and building outlines plus low-detail features. High-detail diagrams showed vegetation and street-level details (e.g., parking) plus medium- and low-detail features. In the expanded version of this paper, we report on a second experiment comparing these levels of detail, finding that medium-detail diagrams produced slightly better results than the others [Kohler et al., 2017]. Therefore, we used medium-detail diagrams in this experiment comparing diagrams to ground-level photos.

3.4 Experiment Design

The study was a between-subjects experiment. The independent variable was reference material with three levels: diagram only, photo only, or both. Location depicted in the diagram was a covariate with three levels: BSB, LAX, or CLT. Therefore, there were nine possible conditions. The dependent variables were the participants' binary judgements on each of the 16 subregions.

We recruited participants from Amazon Mechanical Turk (MTurk). We randomly assigned each worker to one of the nine conditions, and we assigned 60 workers to each condition, for a total of 540 workers. Pilots showed that workers took an average of five minutes to complete the task, so we paid $1.21 per task, reflecting minimum wage in our location for 10 minutes of work. We restricted the task to US-based workers but used no other qualifications.

Task and Procedure

After accepting the task, each participant completed an online IRB consent form and a short, self-paced tutorial. The participant then proceeded to examine each of 16 subregions in the grid and mark it as Yes/Maybe or No. Based on average completion times in pilot studies, we set the time limit at 10 minutes to encourage fast responses. All tasks had exactly one correct subregion and 15 distractors.

We took care to design the crowd task and interface to be as realistic as possible from the worker's perspective. Workers did not know whether their region contained a correct subregion, and received no feedback on their judgements. Therefore, the worker experience would be the same for real-world scenarios where it was unknown whether the region contained a correct subregion.

Data Cleaning and Analysis

In our pilot studies, individual workers showed high variance in task performance. We experimented with different aggregation strategies and found that forming triads (groups of three workers) with a one-yes rule yielded the best results. The one-yes rule means that if at least one of the three workers judged a subregion to be a Yes/Maybe, then it would be categorized as a yes, while only a unanimous judgement of No across all three workers would be categorized as a no. We randomly grouped the 60 workers for each condition into 20 triads per condition in the results that follow.

Next, we compared each triad's judgement to our gold standard to calculate true positives and false positives. We used these measures rather than precision and recall because geolocation is a needle-in-the-haystack problem, where false negatives are much worse than false positives.

We performed statistical analyses in R. Shapiro-Wilk tests showed that the dependent variables failed a normality assumption, so we used Kruskal-Wallis tests as a nonparametric alternative to ANOVAs. We used Dunn's tests to perform post-hoc analyses, with Bonferroni correction to adjust p-values for multiple comparisons.

4 Results

For true positives (Fig. 3), the diagram-only condition performed best, with 98.3% of triads marking the correct subregion. The both condition performed slightly less well with a 90% success rate. Ground photo-only trailed behind with 78.3% of triads finding the correct subregion. Reference material had a significant effect on true positives, χ2 (1) = 4.111, p < 0.05. Post-hoc analysis showed that true positives were significantly higher for diagram-only compared to ground photo-only, z = 3.476, p < 0.01. Also, the both condition performed marginally significantly better than ground photoonly, z = 2.028, p = 0.128. There was no significant difference in true positives for the both condition vs. diagram-only, z = -1.448, p = 0.443.

For false positives, the crowd generally reduced the search area by about half, regardless of reference material. Diagramonly workers produced slightly more false positives (M = 51.3%), followed by both (M = 48%), and then photo-only (47%), but the differences were not significant, χ2 (15) = 12.691, p = 0.626.

5 Discussion

This study investigated how the type of reference material affected the crowd's geolocation performance. We found that the diagram by itself results in significantly higher true positives compared to the ground photo by itself. The diagram allows crowds to achieve near-perfect performance (98.3% of triads found the correct subregion), whereas only 78.3% of triads found it in the ground photo-only condition. Our intuition is that experts would be unlikely to trust a crowd that misses the target one out of every five times, so the ground photo by itself may not be a viable approach.

False positives were around 50% for all conditions and reference material did not have a significant effect. This means that in all cases, the crowd reduced the search area by about half. More importantly, for the diagram-only condition, the search area was cut in half while still including the correct subregion 98.3% of the time. Further, the significance test showed that quality is not a zero-sum game: the diagram condition's excellent true positives do not come at a cost of more false positives.

The above results indicate that the aerial diagram yields a significant improvement in quality for crowdsourced image geolocation. When a triad of workers is shown just the ground-level photo, they miss the correct subregion one out of every five times. When the photo is replaced with an aerial diagram, crowds found the target 98.3% of the time, with no increase in false positives. Crowds also reduce the search area by half. Therefore, the evidence suggests that crowds provided with a diagram could substantially augment an expert's image geolocation process.

6 Conclusion and FutureWork

This paper explored how crowdsourcing could support experts in a geolocation task. We contributed a new diagramming technique, adapted from expert practice, that can be used to help novice crowds more effectively analyze satellite imagery. We also contributed a large-scale crowdsourcing experiment demonstrating the value of our technique. We found that aerial diagrams are significantly better than ground-level photos in supporting crowdsourced satellite image analysis.

As our study locations were all city-based, our results primarily speak to urban geolocation tasks. Our approach may also extend to rural areas, which share many task characteristics with geolocation of urban imagery, but also face some distinct challenges, such as scarcity vs. overabundance of image clues [Mehta et al., 2016].

The approach presented here seeks to minimize expert intervention, but future work is needed to understand how best to integrate the crowd's judgements into an expert's workflow. Other opportunities involve leveraging computer vision tools to support crowds and experts, such as:

• Context identification: Many images on social media do not have surrounding context suggesting a general location. Systems like PlaNet could suggest high-probability sectors to narrow the search space for crowds.
• Diagram generation: Ground-to-aerial systems like Where-CNN could not only suggest potential location matches, but also extract distinctive features to help experts build a diagram more quickly and accurately.
• Image comparison: Sketch recognition systems like Google's Quick, Draw! could compare an expert diagram to satellite imagery and return potential matches.

We propose that hybrid pipelines or mixed-initiative systems composed of crowds, experts, and algorithms, each complementing the others with its unique strengths, offer the greatest potential to support complex image analysis and sensemaking. This paper offers a glimpse of these possibilities in demonstrating how novice crowds can augment the work of experts in geolocation and verification tasks.

Acknowledgments

We thank the study participants, the anonymous reviewers, and members of the Crowd Intelligence Lab. This research was supported by NSF IIS-1527453 and IIS-1651969.