Where Is Your Place, Visual Place Recognition?

1 Introduction

Visual Place Recognition (VPR) is a rapidly growing topic: Google Scholar lists over 2300 papers matching this exact term, with 1600 of them published since the pivotal survey paper by Lowry et al. in 2016. While exhaustive surveys of works on VPR are given elsewhere [Lowry et al., 2016; Zhang et al., 2020; Masone and Caputo, 2021], our goal here is to lay a concrete understanding of VPR as a research problem. VPR capability is based on the fundamental ability to aptly represent incoming information and associate the incoming information with previously stored information. This is required for an embodied or augmented agent to intelligently understand the operating environment, to navigate within it and interact with it, thus evolving Simultaneous Localization And Mapping (SLAM) into Spatial AI [Davison, 2018] 1 . We argue that research on VPR has increasingly become more dissociated: there is no standard definition of a 'place' and comparison of methods is challenging as benchmark datasets and metrics vary substantially.

* Sourav Garg and Tobias Fischer contributed equally to the paper. 1 SLAM is the problem of incrementally estimating the envi-

ronment's structure while simultaneously tracking the robot's pose

In light of the dissociation and while retaining the accessibility of a compact treatment, we discuss VPR with regards to its definition (Section 2), how it closely relates to other areas of research (Section 3), what the key drivers of VPR research are (Section 4), how to evaluate VPR solutions (Section 5), and what key research problems still remain unsolved (Section 6). Figure 1 illustrates the outline of our paper and shows how the various sections are interrelated.

2 What is Visual Place Recognition?

Lowry et al. state that VPR addresses the question of "given an image of a place, can a human, animal, or robot decide whether or not this image is of a place it has already seen?" [Lowry et al., 2016]. One can easily see that such a capability is of

within the environment. Traditionally SLAM's purpose was purely navigation, while the inclusion of semantics moved SLAM towards Spatial AI, whose aim is intelligent goal-driven interaction of the robot with the environment and other agents [Cadena et al., 2016].

crucial importance in tasks like localization and navigation, which in turn become ever more important with the advent of artificial intelligence (AI) in autonomous cars, mobile robots that interact with humans, and intelligent augmentation platforms such as the HoloLens 2. Inspiration for VPR is often drawn from the animal kingdom, given the remarkable localization and navigation capabilities of even "simple" animals, and the relatively well understood underlying mechanisms (even leading to 2014's Nobel Prize for the discovery of place cells and grid cells, as further detailed below).

Although it might seem inevitable to define a place first, we instead define VPR directly, remembering that it is a comparison of visual data, observed from same or different physical locations with same or different viewpoints. We argue that a reference and query observation can lead to successful recognition if there exists a certain degree of visual overlap 2 due to overlapping field-of-view of the underlying sensor, whereby the acceptable degree depends on the drivers introduced in Section 4. This implies that: 1) being at the same physical location is not sufficient, the orientation (i.e. viewpoint) needs to be somewhat similar as well, and 2) places can also be recognized when observed from distant physical locations (see Figure 2). In short, we define VPR as the ability to recognize one's location based on reference and query observations perceived from overlapping field-of-views. Note that our definition requires rethinking the typical notion of localization threshold (as used by almost all datasets and evaluation metrics, see Section 5), which is based on metric distances without considering orientation.

Our definition is complementary to that of Lowry et al., but has a different underlying motivation. Lowry et al.'s definition is in line with the notion of place cells, which fire when an animal is in a particular place in the environment, irrespective of the animal's orientation. Instead, our definition is in line with spatial view cells, which fire when a specific area of the environment is gazed at by the animal, irrespective of the particular location [Georges-Franc¸ois et al., 1999] (see Figure 2).

2 Here visual overlap is defined as the common visibility of 3D points/regions in 2D image views. We do not consider VPR based on prior knowledge, e.g. scene completion [Hays and Efros, 2007; Song et al., 2017], where visual overlap may not be needed.

We note that in the context of robotics, VPR often involves sequential imagery (i.e. using multiple images that were captured over tens or hundreds of meters) rather than single images, as this can significantly improve place recognition performance, especially so for challenging environments [Lowry et al., 2016]. For such sequence-based methods, the equivalent of visual overlap is the overlap of the volume spanned by the sequence.

3 Related Areas

In this section, we highlight similarities and differences of VPR with a handful of related areas of research. This section does not include downstream tasks such as visual SLAM, which are instead covered in Section 4.3 (see also Figure 1). While the relation to image retrieval has been discussed in other works [Lowry et al., 2016; Zhang et al., 2020; Masone and Caputo, 2021], it is for the first time that VPR's relation to video retrieval, visual landmark recognition, and overlap detection is systematically presented. We argue that for each of those areas, there is a potential for mutual benefits: research into VPR can offer insights for these areas and vice versa.

Image Retrieval: Image retrieval refers to the general problem of retrieving relevant images from a large database [Arandjelovi´c et al., 2017]. VPR is commonly cast as an image retrieval problem that involves a nearest neighbor search of compact global descriptors [Arandjelovi´c et al., 2017] or crossmatching of local descriptors [Liu et al., 2021; Hausler et al., 2021; Tourani et al., 2021]. With regards to solving the nearest neighbor search problem, VPR and image-retrieval systems face similar challenges. However, the underlying goals differ between the two areas. For image retrieval, similarity criteria could be based on semantic categories such as 'clothes' as a product category or nighttime image as an environmental condition category. However, with the additional context of being a 'place' (see Section 2), VPR deviates from the process of merely retrieving a "similar" image, which instead is one of the challenges of VPR and referred to as perceptual aliasing (see Section 4). The notion of similarity in VPR is constrained to matching spatial information, where images captured from the same place would be considered a true match even if environmental conditions are dissimilar (e.g. day vs night).

Video Retrieval: The video retrieval problem is analogous to image retrieval, except that relevant videos are retrieved from the database leveraging sequential information. In VPR literature, sequence-based VPR is typically implemented as a decoupled approach, where single image-based retrieval is followed by sequence score aggregation [Ho and Newman, 2007; Milford and Wyeth, 2012]. The recent introduction of explicit sequence-based place representations (where the representation itself describes the sequence), posing VPR as a video retrieval problem, opens up new opportunities to obtain solutions robust to extreme appearance variations [Garg and Milford, 2021; Garg et al., 2020a; Arroyo et al., 2015].

Visual Landmark Recognition and Retrieval: Visual landmark recognition is the classification problem of given an image and a set of images belonging to a large variety of landmarks, deciding to which landmark this image belongs. Recently, the Google-Landmarks dataset [Weyand et al., 2020] presented a new large-scale instance-level recognition and retrieval challenge, with the number of landmarks 3 increased from 30,000 to 200,000 in its second version. This large-scale recognition is an extreme classification problem [Choromanska et al., 2013], where existing recognition solutions have relied on retrieval (nearest neighbor search) [Teichmann et al., 2019]. Google-Landmarks comprises specific places (with the semantics of unique proper names) as opposed to general place categories (with the semantics of common names) [Zhou et al., 2017; Wu et al., 2009].

In contrast, VPR refers to the ability of distinctively recognizing any ordinary place or a region in the 3D world, thus posing an 'extremer' classification problem. It remains to be seen how methods developed for landmark recognition and retrieval can be leveraged in the context of VPR - recent advances include learning to aggregate 'relevant' landmarks [Teichmann et al., 2019], as well as jointly training local and global descriptors [Cao et al., 2020; Sarlin et al., 2019].

Visual Overlap Detection: As discussed in Section 2, our definition of VPR is based on an overlapping field of view between the two places that should be matched; thus VPR and the area of visual overlap detection become more closely linked. The contrast between defining VPR using visual overlap as opposed to "positional offset" impacts the choice of ground truth for both training and evaluation procedures. This contrast has recently been shown to lead to noticeable changes in absolute performance when benchmarking localization algorithms [Pion et al., 2020].

As the ground truth visual overlap might not be available for all datasets, overlap detection measures could be used as a supervision signal [Rau et al., 2020; Chen et al., 2020] to develop better VPR techniques. A noteworthy recent proposal on overlap detection [Rau et al., 2020] introduced the 'normalized surface overlap' to measure the number of pixels of image A visible in image B (and vice versa). This leads to an asymmetric, but interpretable, measure that can also estimate the relative scale between pairs of images.

3 In the context of mobile robotics, the term 'landmark' is typically used to indicate any specific visual entity in the scene relevant for localization [Luo et al., 1992; Xin et al., 2019].

4 What Drives VPRResearch?

This section outlines the three key drivers of spatially intelligent systems, including intelligent autonomous systems operating in industry and household domains. As drivers, we refer to components that typically impose requirements on the system with regards to a) how the problem should be defined, b) how the solution (in the context of VPR: place representation and matching) should be designed, and c) how these solutions should be evaluated, both in terms of datasets and metrics. The three drivers are the Environment where an agent operates (Section 4.1), the Agent on which the spatially intelligent system is deployed (Section 4.2), and the Downstream Task that is performed (Section 4.3). In practice, different aspects of each of these drivers are simultaneously at play. We detail why it is crucial to understand the influence of these drivers to design better spatially intelligent systems, in particular in the VPR domain.

4.1 Environment

The first driver of VPR research is the operating environment, where research often branches out, as methods that work in certain environment types might cease to do so in other environment types. Differing branches include indoor vs outdoor, suburban vs highway, structured vs open, and human-made vs natural. The operating environment is often tightly coupled with the robotic agent choice (Section 4.2) - for example, driverless cars do not operate in office environments, or at least should not.

While the general aim of VPR systems is often stated to be invariance to changes in viewpoint as well as changes in appearance (including structural, seasonal, and illumination changes) [Lowry et al., 2016; Arandjelovi´c et al., 2017; Garg et al., 2018b; Zhang et al., 2020], we argue that 1) not all environments/agents require invariance to both viewpoint and appearance (as detailed below), and 2) that there is a trade-off between viewpoint and appearance invariance achievable by current systems (as detailed in Section 6.1). Therefore, knowing the operating environment can provide crucial advantages when deciding how to represent and match places.

Structured Environments and Viewpoint Variations: In well-structured environments such as road infrastructure, the extent of 6-DoF viewpoint variations is generally confined, e.g. for driverless cars on roads, viewpoint varies mostly in the yaw direction [Maddern et al., 2017]. Similar effects in viewpoint variations can be observed for other platforms, too. For example, as soon as aerial vehicles reach a certain height, one can assume a planar homography ("flat world"), simplifying template matching [Saurer et al., 2016; Mount et al., 2019]. Planar homographies are also present when mounting the camera at a fixed distance from the surface and pointing towards the surface. In structured indoor environments such as warehouses and offices, aisles and corridors enable Manhattan world assumptions and often simplify Simultaneous Localization And Mapping (SLAM) [Li et al., 2018].

Environment-Dependent Appearance Variations: Appearance invariance is similarly often constrained when assuming a certain operating environment. However, this kind of invariance is harder to quantify as changes in appearance can originate from a wide range of factors: Examples include changes in the time of the day, seasonal changes, structural changes, and weather changes. Therefore, while viewpoint change could be quantified by the metric shift in translation and rotation, there is no linear scale in the difficulty of appearance invariance [Zaffar et al., 2021]. There are even some counter-intuitive examples, where a reference image captured outdoor in the morning might be easier to match to a well-lit nighttime image than to an image captured at noon which has shadows cast on a large area of the image [Corke et al., 2013].

For different platforms and environments, the requirement of representing and matching places in an appearance-invariant or viewpoint-invariant manner can differ significantly. For example, driverless cars typically traverse a well-defined route and could trade-off viewpoint-invariance with appearanceinvariance which can be relatively more challenging due to variations in the time of day, season, structural changes including roadworks and differing traffic conditions [Warburg et al., 2020]. On the other hand, when an autonomous agent is deployed indoors or when considering an unmanned aerial vehicle, its route or maneuvers may not always be constrained, thus requiring viewpoint-invariance more than appearanceinvariance.

Perceptual Aliasing: Another consideration that is tied to the operating environment is the extent of perceptual aliasing. Perceptual aliasing is the problem that two distinct places can look strikingly similar, often more similar than the same place observed under different conditions [Lowry et al., 2016]. For example, indoor environments often contain corridors and hallways that are hard to distinguish. In outdoor settings, different places along a highway or a natural vegetative environment tend to be more perceptually aliased than different places within the man-made urban or suburban dwellings.

Dynamic Environments: Problems related to the operating environment that - to our knowledge - have not yet been addressed in VPR research are sensor dust, reflections (in glass or puddles) and undesired objects close to the camera (e.g. windscreen wipers). Such conditions are expected in challenging environments like mines and forests, which have come into focus in recent years [Nardari et al., 2020; Garforth and Webb, 2020]. It would be interesting to model the impact of such 'noise' explicitly or measure the impact of sensor noise in existing VPR systems.

4.2 Agent

VPR has widespread applications and is thus deployed on a large variety of robotic platforms, including unmanned ground vehicles and autonomous cars [Doan et al., 2019], unmanned aerial vehicles [Zaffar et al., 2019] and unmanned underwater vehicles [Li et al., 2015b]. Other platforms where VPR is applied are those tightly coupled to human users such as virtual/augmented reality devices [Sattler et al., 2016] and mobile phones [Torii et al., 2018].

Computational Resources: A robotic agent typically runs a large number of processes, many of them interacting with each other through tools like the Robot Operating System (ROS) [Quigley et al., 2009; Fischer et al., 2021]. These processes share limited onboard resources and often require cognitive architectures [Fischer et al., 2018] to interact efficiently. Thus the resources dedicated to the VPR system might be relatively small, and a GPU (that significantly boosts inference times of deep networks) might not be available. Similarly, storage limitations could mean that the reference map of the operating environment (in the form of images, global/local descriptors, point clouds) has to be of reasonable size. Section 6 discusses some of the open problems in VPR in this context, for example, compact global description, efficient indexing and quantization, and hierarchical place matching pipelines.

Suitable Sensor Suite: Depending on the agent and the operating environment, robust VPR solutions can be developed by using additional suitable sensors. For example, event cameras perform exceptionally well when a high dynamic range is required, such as when exiting a dark tunnel and moving into bright sunlight [Fischer and Milford, 2020]. LIDAR-based systems can perceive the scene's geometry even in the most challenging nighttime conditions, although those systems lack appearance information [Guo et al., 2019]. Using omnidirectional cameras or multi-camera rigs increase the field-of-view and thus the visual overlap, which results in reduced complexities in image matching.

Correct sensor type choice can also aid in tackling specific challenges such as nighttime conditions. Crucially, the sensor capabilities should drive the research regarding what characteristics are required in our learned descriptors. We believe that using novel sensor types such as 3D ultrasonic sensors (e.g. the Toposens TS3) and sensor fusion [Jacobson et al., 2015] could further improve the robustness of VPR systems.

While some sensors can be a replacement for RGB cameras, another area worthy of more thorough investigation is the use of additional information in the form of prior position or egomotion. For example, one can assume that autonomous cars are equipped with a GPS sensor. Still, despite the popularity of datasets such as Oxford RobotCar [Maddern et al., 2017] that contain GPS information, it is rarely used for VPR [Vysotska et al., 2015] - using GPS information in environments where available could refocus research on GPS-denied environments like tunnels or underground mines that have distinct challenges. While there are many examples of GPS-denied environments, almost all mobile robots have some odometry information, but it has only been used in a limited manner for VPR [Pepperell et al., 2014].

4.3 DownstreamTask

Here, we consider the different tasks that an agent (robotic platform or intelligent augmentation) might perform. In this paper, we consider the VPR problem in the context of localization, although VPR does not strictly imply localization. For example, VPR can also be used to solve the problem of scene change detection where localization based image pairing is assumed [Park et al., 2021], especially in the context of updating the map of the environment [Alcantarilla et al., 2018; Tanaka, 2018].

It is also worth noting that localization does not strictly require VPR. For example, satellite-based (e.g. GPS) coarse position estimation can be fused with additional sensor information to enable 6-DoF visual SLAM in outdoor environments [Schneider et al., 2016]. However, limitations of satellite-based pose estimation such as unavailability indoors, sensitivity to atmospheric changes, and signal obstruction in cluttered environments, deem vision-based alternatives such as VPR necessary.

Localization, SLAM and Kidnapped Robot Problem: A purely topological visual SLAM system 4 can be directly defined through VPR, which is highly relevant for largescale mapping [Cummins and Newman, 2008; Doan et al., 2019]. Such a topological SLAM system requires determining whether the currently observed place is a revisited one or is a new 'unseen' place, thus posing unique design requirements on VPR.

For metric or topo-metric SLAM systems 4 , the final goal is typically to estimate the agent's 6-DoF pose, thus requirements of VPR systems vary in this case. For example, when VPR is used to provide a coarse estimate of the pose within a 6-DoF localization-only [Toft et al., 2020] or SLAM algorithm [MurArtal et al., 2015], the error bounds need to be very tight and the visual overlap between the two places relatively large with sufficient parallax. This is opposed to a scenario where loose error bounds are sufficient - a rough location estimate might sufficiently narrow down the search space for a subsequent laser-based pose estimation for global re-localization of a mobile robot ("kidnapped robot problem") [Jacobson et al., 2021].

The requirements with regards to precision and recall are also varying for different scenarios. When using VPR for generating loop closures for SLAM (i.e. recognizing that a location has been visited previously, so that a globally consistent map can be built), incorrect matches can lead to catastrophic failures, thus requiring high precision VPR [Cadena et al., 2016]. On the other hand, one could use VPR to select top k matches which are then passed to computationally more intensive stages for further processing; in this case, higher recall is more important than the precision. Thus, the downstream task is a key determining factor for formulating and evaluating VPR, as further discussed in Section 5.

Higher-level Tasks: The requirements of some downstream tasks like SLAM and Structure from Motion (SfM) are relatively well understood; yet, these requirements are very distinct and probably need a suitably tailored treatment. For example, SfM-based large-scale 3D reconstruction is typically performed offline [Sch¨onberger and Frahm, 2016] and needs sub-pixel accurate alignment of images. The computational requirements of a VPR system then play a much lesser role than in real-time deployments on a mobile platform mapping an unknown environment using visual SLAM.

4 Topological SLAM captures the connectivity of the environment rather than building a geometrically accurate map [Brooks, 1985]. This is opposed to metric SLAM, where geometrically accurate maps are built. Topo-metric SLAM systems build local maps that are geometrically accurate and then topologically connected to form the overall map [Cadena et al., 2016].

The requirements of other "higher-level" tasks such as those of augmented reality platforms and navigation are not yet well established. This is in part due to the complex hierarchical nature of typical spatially intelligent systems, for example an augmented reality platform would involve many interrelated components such as image retrieval, sequential localization, local feature matching, visual odometry, and pose refinement [Stenborg et al., 2020]. Furthermore, the utility of VPR and mapping for navigation purposes [Milford and Wyeth, 2007; Dall'Osto et al., 2020] is a vastly unexplored area, and a deeper understanding of task requirements is needed.

5 How to Evaluate Visual Place Recognition?

This section discusses the evaluation datasets and metrics, in the context of the drivers.

5.1 Evaluation Datasets

There are numerous place recognition datasets, each covering different aspects of VPR (for recent overviews see [Warburg et al., 2020; Masone and Caputo, 2021]). Thus some datasets are better suited to investigate specific configurations of proposed drivers (i.e. environments, agents and downstream tasks, see Section 4), while other datasets better represent different scenarios. This highlights the importance of clearly stating the application scenario targeted by a particular VPR system - it may be sufficient that the VPR system excels in datasets that are close to the actual use-case (but not in others).

Recent progress has enabled easier comparison of different methods. VPR-Bench [Zaffar et al., 2021] provides a mechanism for the comparison of a new method on an extensive range of datasets. In the light of highly successful standard benchmark datasets in other research areas like visual object tracking [Kristan et al., 2018], we believe that such benchmarking will accelerate VPR research. Mapillary Street Level Sequences (MSLS) [Warburg et al., 2020] is a single dataset that tries to capture all variations of appearance/viewpoint change at once. MSLS notably also introduces 'sub-tasks' that can be separately investigated, including sub-tasks like summer to winter, day to night, and old to new. An additional benefit of MSLS is that it provides a hold-out test set that can be used for challenges.

If the aim is to design a VPR system applicable in all different scenarios, an open challenge is to design systems that are equally applicable indoors and outdoors. Few studies evaluate systems both indoors and outdoors, one of them being the above mentioned VPR-Bench [Zaffar et al., 2021]. VPRBench has shown that performance trends can vary noticeably across environment types, e.g. indoor vs outdoor. However, care should be taken to not make generic assumptions about an architecture when the trained descriptors heavily depend on the training data - the training data should be representative of the data encountered at deployment time. Most recently, [Warburg et al., 2020] have shown that training on more diverse data drastically improves performance on unseen data. This is distinct from the approach where different network configurations are explicitly trained for different scenarios, e.g. one for indoors and another for outdoors [Sarlin et al., 2020].

5.2 EvaluationMetrics

The previous examples show that the downstream task and the relevant evaluation metrics are tightly coupled. However, we note that many VPR papers do not state why a particular evaluation metric was chosen. Notable exceptions include system papers where VPR is one of many components, and a specific downstream task is considered, such as [Cummins and Newman, 2008]. The computer vision community typically uses the Recall@k measure, which indicates that in this context the VPR system is benchmarked based on its ability to retrieve a correct match within the top-k retrievals regardless of the false matches. On the other hand, the mean average precision (mAP) metric [Philbin et al., 2007], used in the image retrieval community, explicitly penalizes selection of false matches. The mAP metric could be adopted to measure VPR performance for SLAM-like downstream tasks (Section 4.3) where precision is more important, complementing measures like Recall at 100% Precision.

The area under the precision-recall curve and the F-score are sometimes used as summary statistics [Molloy et al., 2021]. However, their practical use is unclear, as these summary statistics imply that recall and precision are of similar importance, which is unlikely the case for most downstream tasks. Moreover, these measures are based on the distribution of match scores which may only be relevant for topological SLAMlike scenarios where VPR needs to be highly precise and no subsequent outlier rejection method is employed.

Most of the VPR datasets in robotics are in the form of trajectories with inherent sequential information (Section 2). Thus, evaluation metrics such as 'maximum open-loop distance traveled' (that is, the extent of visual odometry or dead reckoning based robot motion without loop closures) have also been considered in the literature [Clement et al., 2020]. We believe it would be beneficial to investigate metrics that tightly couple single-image and sequence-based VPR.

6 What Are Open Research Problems?

This section aims to highlight open research problems, considering the drivers discussed in Section 4. For space reasons and to avoid duplication, we do not cover the open research problems discussed in recent surveys on deep learning methods for VPR [Zhang et al., 2020; Masone and Caputo, 2021], which include using autoencoders as an alternative to Convolutional Neural Networks (CNNs), use of generative methods including Generative Adversarial Networks (GANs), using semantic information, making use of heterogeneous data including multi-sensory fusion, and the choice of loss function.

Here, we broadly classify the open research problems into: 1) representation, discussing the need for better global descriptors and enriched/synthesized reference maps, and 2) matching, discussing the need for better hierarchical matching frameworks, relevant distance metrics and 'learning to match'.

6.1 Place Representation

Global Descriptors - Appearance & Viewpoint Invariance: Section 4.1 discussed the requirements on viewpoint and appearance invariance depending on the operating environment. Here we note that there is a trade-off when learning a descriptor of a fixed size/type: increasing viewpoint-invariance will inevitably reduce some degree of appearance invariance (assuming the same amount of training data) [Arandjelovi´c et al., 2017; Chen et al., 2017; Garg et al., 2019]. This is evident from significant differences observed in place recognition performance when considering a cross-combination of datasets such as Nordland (same view, varying appearance) [S¨underhauf et al., 2013] and Pittsburgh (similar appearance, varying view) [Arandjelovi´c et al., 2017] with feature learning/aggregation methods such as HybridNet (viewpoint-assumed) [Chen et al., 2017] and NetVLAD (viewpoint-agnostic) [Arandjelovi´c et al., 2017].

There is a vast research gap, and a need for global description mechanisms that go beyond the binary nature of encoding viewpoint, that is, viewpoint-assumed vs viewpoint-invariant. This might be achieved by learning novel ways to incorporate local geometric information in the global descriptor formation, such as using vertical blocks (Stixels) [Hernandez-Juarez et al., 2019], semantic blobs [Gawel et al., 2018], objects [Qin et al., 2021] or superpixels [Neubert et al., 2015], where learning could be based on attention mechanisms such as that employed in Transformers [Vaswani et al., 2017] and Graph Neural Networks [Veliˇckovi´c et al., 2018].

Global Descriptors - Efficiency: Most of the state-ofthe-art global image descriptors are high-dimensional (with dimensions varying from 512 [Sarlin et al., 2019] to 70,000 [S¨underhauf et al., 2015a]). Increasing the descriptor's dimensionality directly leads to increased computational requirements. To improve efficiency, researchers commonly employ dimension-reduction methods such as Principal Component Analysis (PCA), and have also explored quantization [Jegou et al., 2010; Brandt, 2010; Ge et al., 2013], binarization [Lowry and Andreasson, 2018; Arroyo et al., 2015; Jegou et al., 2008], hashing [Vysotska and Stachniss, 2017; Gionis et al., 1999; Andoni and Indyk, 2006] and efficient indexing [Cao et al., 2020] techniques.

However, there have not been any attempts to learn these efficiency-inducing processes for VPR, particularly considering that retrieving places can include additional information in the form of sequences or odometry. This could be achieved by learning to reduce dimensions [McInnes et al., 2018; Amid and Warmuth, 2019] or to hash [Wang et al., 2017], while maintaining the overall structure of the appearance space learnt through the existing global descriptor methods. Existing efficient VPR techniques consider sequential or odometry information in a decoupled manner [Vysotska and Stachniss, 2017; Garg and Milford, 2020] but could benefit from jointly considering additional information when optimizing for efficiency.

Enriched Reference Maps: With the rapid increase in data gathering, more so in the field of autonomous driving, it is high time to consider the use of an enriched reference map, which could be in the form of multiple reference images per location [Churchill and Newman, 2012] or semanticallyannotated 3D maps [Garg et al., 2020b]. In the simplest case, choosing the best reference set can lead to vast performance improvements. More sophisticated approaches fuse multiple reference sets to achieve even better performance [Churchill and Newman, 2012; Linegar et al., 2015;

Molloy et al., 2021]. Multiple reference sets are often used when long-term autonomy is required, as structural changes can be detected and incorporated over time. While significant progress using multiple reference maps has been made in the past, open questions remain around the increased storage and computational requirements when using multiple reference sets. There is clearly a trade-off, and preliminary efforts in that direction [Doan et al., 2019] need further attention.

View Synthesis: To deal with significant viewpoint variations, researchers have also explored matching through multiple synthesized views of the observed places [Torii et al., 2018; Milford et al., 2015], although the process requires additional computation. However, this can be mitigated by performing view synthesis offline during the mapping traverse and using compact global descriptors with an efficient nearest neighbor search for retrieval. The current bottleneck of automatically generating accurate and relevant views can potentially be addressed by recent advances in volumetric rendering [Mildenhall et al., 2020], performed offline to generate novel views under novel lighting conditions.

6.2 Place Matching

Mutually-informed Hierarchical Systems: Different downstream tasks can impose very different requirements on the VPR system (see Section 4.3). For example, a visual SLAM system can be built with [Cummins and Newman, 2011] or without [Angeli et al., 2009; Cummins and Newman, 2008] using a geometric verification step based on local feature matching. Cummins and Newman found this verification to be particularly essential for large datasets. However, it remains an open question how an effective hierarchical system should be designed, where the variables are: the number of complementary VPR techniques fused [Hausler et al., 2019], number of stages, and types of unique methods involved, e.g., query expansion [Chum et al., 2011] and keypoint filtering [Garg et al., 2018b].

The complementarity and information transfer within different stages in the hierarchy requires an in-depth investigation. This can reveal answers to several overarching questions: should these stages always operate independently or could earlier stages better inform subsequent stages (beyond simply providing candidate images); could one selectively apply a subset of the techniques to save computational resources; how such behavior of hierarchical retrieval can be learnt; and in doing so do some of the stages become redundant.

Choice of Distance Metric: When comparing the global descriptors of two images, one has to choose a suitable distance metric or a similarity measure. Some of the most commonly employed measures include Euclidean [Arandjelovi´c et al., 2017; S¨underhauf et al., 2015b], cosine [S¨underhauf et al., 2015a; Garg et al., 2018a; Garg et al., 2018b], and Hamming [Lowry and Andreasson, 2018; Neubert et al., 2019] distance. While some descriptors are better matched using one distance than the other, the range of distances distribution is typically relatively narrow, even in non-matching images of a completely different appearance. Therefore a more systematic investigation considering both a theoretical viewpoint and practical performance implications is needed. In particular, important consideration factors include suitability to loss functions (e.g. max-margin triplet loss) [Arandjelovi´c et al., 2017; Revaud et al., 2019; Garg and Milford, 2021], descriptor normalization [Arandjelovic and Zisserman, 2013; Garg et al., 2018a; Schubert et al., 2020], whitening [J´egou and Chum, 2012; Arandjelovic and Zisserman, 2013], feature scaling [Beatty and Manjunath, 1997; Li et al., 2015a], and quantization/binarization [Jegou et al., 2010; Lowry and Andreasson, 2018].

Learning toMatch: While learning to 'describe' (i.e. local or global descriptors) has been widely explored, there have been limited attempts to learn to 'match'. Such matchers can either be learnt through siamese networks [Altwaijry et al., 2016] or cross-attention based on graph neural networks [Sarlin et al., 2020]. The outcome of such matcher could either be a matched reference index or relative 3D pose [Gridseth and Barfoot, 2020]. Learning-to-match frameworks for VPR and localization could potentially eradicate the need for sophisticated matching pipelines.

7 Conclusions

This survey defines VPR based on the visual overlap of two observations, in line with spatial view cells found in primates. This new definition enabled us to discuss how VPR closely relates to other research areas. This paper also identified and detailed three key drivers of Spatial AI: the environment, agent and downstream task. Considering these drivers, we then discussed numerous open research problems that we think are worth addressing in future VPR research.

To date, VPR research has addressed the problems of representing, associating (matching), and searching of spatial data, and is a key enabler of Spatial AI. Further advances in VPR research will require unifying the efforts of the artificial intelligence, computer vision, robotics, and machine learning communities, particularly taking into account embodied agents. To achieve this, an in-depth understanding of the problem, research goals and evaluation protocols is necessary, and this paper takes a step in that direction.

Acknowledgments

This work received funding from the Australian Government, via grant AUSMURIB000001 associated with ONR MURI grant N00014-19-1-2571. The authors acknowledge continued support from the Queensland University of Technology (QUT) through the Centre for Robotics.