RoLocMe: A Robust Multi-agent Source Localization System with Learning-based Map Estimation

1 Introduction

Source localization is a well-established research problem with numerous applications, including forest fire tracking [Sharma et al., 2020], dangerous gas leakage detection [Lu et al., 2014], radioactive material localization [Morelande et al., 2007; Zhao et al., 2022], search-and-rescue operations [Atif et al., 2021], and security surveillance [Rybski et al., 2000].

Recent deep learning-based source localization methods [Zhang et al., 2016; Zubow et al., 2020; Zhan et al., 2021; Wang et al., 2022; Lin et al., 2022; Raj and B. S., 2023] learn directly from data and often outperform traditional techniques, especially under imperfect conditions such as sparse sensor coverage or signal degradation, where classical approaches tend to falter. However, extreme signal attenuation challenges both. Although traditional methods frequently fail outright, deep learning models, though affected, tend to be more robust, leveraging weak signal patterns to maintain some level of performance. Mobile sensors address this shared vulnerability by actively seeking higher-quality data to reduce uncertainty and improve localization accuracy.

Early research on mobile sensing for source localization primarily relied on path-planning strategies [Koutsonikolas et al., 2006; Hu et al., 2008; Jiang et al., 2011; Rezazadeh et al., 2014], which were straightforward to implement but struggled to generalize in complex environments. To overcome these limitations, recent efforts have shifted toward data-driven reinforcement learning approaches, broadly categorized into single-agent and multi-agent methods. Singleagent approaches [Wu, 2019; Ebrahimi et al., 2021; Proctor et al., 2021; Zhao et al., 2022] have demonstrated adaptability in various environments but often result in prolonged searches and lack generalizability for multi-agent scenarios. Recent multi-agent reinforcement learning studies [Alagha et al., 2022; Alagha et al., 2023; Wickramaarachchi et al., 2024] have shown promise but face challenges in addressing complex environments and severe signal attenuation.

1.1 Our Contributions

This paper presents a multi-agent approach to source localization, addressing two critical challenges: (1) Adaptability in complex environments, where agents must handle multi-path fading in unknown environments with dense obstacles, and (2) Extreme signal attenuation, where latent information from weak signal samples is difficult to discern, posing challenges for effective decision-making. To overcome these challenges, we make the following contributions:

• We propose RoLocMe, a novel system that integrates a radio map estimation module, with the Value Decomposition Network (VDN) [Sunehag et al., 2017] for agent training. Unlike recent methods relying solely on sampled signals and agent positions, an explicit signal propagation map estimation module is seamlessly integrated with a reinforcement learning process, augmenting previous observations to improve navigation and planning.
• We evaluate RoLocMe across various scenarios involving complex environments and extreme signal attenuation and test the system by placing agents at significant distances from the transmitter. Our system achieves at least a 20% improvement in success rate and a 20% reduction in both localization time and the number of actions taken. We further examine the system's scalability by training on various team sizes, demonstrating strong performance across all configurations.
• We conduct a drop-one ablation study by removing individual modules from RoLocMe. The study showed that λ-returns are important in dense reward. And the radio map estimation component proves crucial to the system, boosting agents' performance by at least 15% across different scenarios.

2 RelatedWork

Our work is most related to mobile sensing and agent coordination for source localization, including path planning-based and reinforcement learning-based approaches.

2.1 Path Planning-based Approaches

Early studies on source localization focused on pathplanning, where an agent follows a fixed trajectory-such as Hilbert curves, spiral curves, LMAT trajectories, or Z-curves [Koutsonikolas et al., 2006; Hu et al., 2008; Jiang et al., 2011; Rezazadeh et al., 2014]-until reaching the target or within a threshold. Although these deterministic paths are straightforward to implement, they often lack flexibility and can be challenging to optimize in complex environments.

2.2 Reinforcement Learning-based Approaches

Recent research has shifted toward data-driven reinforcement learning (RL) approaches, which are commonly categorized into single-agent and multi-agent methods.

Single-Agent Approaches

A study introduced multimodel Q Learning, which employs a tabular Q-learning algorithm with pattern recognition on sampled measurements to locate the source [Wu, 2019]. Another approach, PC-DQN [Zhao et al., 2022], applies DBSCAN to sampled measurements before using Deep Q-learning (DQN) [Mnih, 2013], thus enabling efficient searches and demonstrating strong transferability to related tasks. Other work involves training an agent on a predefined trajectory and then fine-tuning its policy with DQN [Ebrahimi et al., 2021]. Meanwhile, source measurements can be transformed into two-dimensional matrices and processed by Double Deep Qlearning [Liu and Abbaszadeh, 2019]. Finally, RAD-A2C combines Proximal Policy Optimization (PPO) [Schulman et al., 2017] with a recurrent neural network to refine search policies [Proctor et al., 2021].

Multi-Agent Approaches

The OMDTL method introduces a novel observation design and a team-based reward function, relying on PPO to foster cooperative agent behaviors [Alagha et al., 2022]. Building on OMDTL, two enhanced methods, MDRL SR and MDRL DC [Alagha et al., 2023], both employ an autoencoder for environment compression to augment observations during PPO training, thereby improving agents' performance. While MDRL SR uses dense rewards, MDRL DC leverages expert demonstrations under sparse rewards. However, neither MDRL SR nor MDRL DC addressed complex environments, and the agents' observation can be further simplified. Another approach proposes multiple reward functions and employs a similar PPO-based framework for multi-agent source localization [Wickramaarachchi et al., 2024].

Our work focuses on multi-agent source localization. We primarily compare our approach with [Alagha et al., 2023] due to the similarity of the problem setting. In contrast, while [Wickramaarachchi et al., 2024] tackles a comparable multiagent task, their sensing strategy allows agents to sample multiple values in a small area around them rather than relying on individual readings at specific positions, making a direct comparison less applicable to our setup.

3 Proposed RoLocMe System

This section introduces RoLocMe's system design, integrating a multi-agent reinforcement learning strategy with SkipNet, a deep learning radio signal prediction model.

3.1 ProblemDefinition andModeling

Consider a geographic Region of Interest (RoI) discretized into a two-dimensional grid of H × W, where H and W are the height and width of the grid, respectively, with (H,W ∈ Z+). A set of agents is deployed on random cells throughout this RoI. Each agent can choose from nine possible actions at each time step: idle or move between cells in one of the eight directions (up, down, left, right, or any of the four diagonals).

Additionally, each agent can observe Received Signal Strength (RSS) at its current cell, know the whole environment, and communicate with other agents. The goal is to efficiently locate and approach the unknown transmitter within a distance of x cells.

Based on the defined problem above, we model it as Partially Observable Markov Decision Process (POMDP) [Oliehoek et al., 2016] because the agents cannot observe RSSs of the entire environment and can only rely on the limited samples based on itself and other agents. POMDP can be described as a tuple < N , S, A, ⊕, P, r, γ >, where:

• N = {1 , ..., n} denotes the set of n agents (robots) where N ≥ 1.
• S = {s1, ..., sk}k∈Z+ denotes the set of finite states.
• A = A×A× · · · ×A ︸ ︷︷ ︸ n is the set of joint actions, where

A is the set of actions for each agent to choose, while at = (a1t , ..., ant ) ∈ A denotes a joint actions at time step t.

• ⊕ = Φ×Φ× ... ×Φ ︸ ︷︷ ︸
• n is the set of joint observations,

where Φ is the finite set of observations of each agent, and ϕt = (ϕ1t , ..., ϕnt ) ∈ ⊕ is a joint observations at time step t.

• P(st+1, ϕt+1|st, at) ∈ [0, 1] denotes the probability that at step t, taking joint actions at in state st results in a transition to state st+1 and joint observations ϕt+1.
• R(s, a) : S ×A → R is a reward function that outputs the team reward when taking joint action a at state s. R(s, a) is from hence forward referred to as rt for the rest of the paper.
• γ ∈ [0, 1] is the discount factor.

At each time step, every agent i observes ϕit ∈ Φ and selects an action ait ∈ A according its policy, denoted by πi . Once all agents have acted simultaneously, they collectively receive a team reward rt. Each agent's objective is to maximize the team's expected cumulative reward over the entire episode.

3.2 RoLocMe SystemOverview

RoLocMe addresses source localization by integrating two deep-learning models: SkipNet [Locke et al., 2023] and the VDN [Sunehag et al., 2017]. SkipNet is a pre-trained signal map estimator that enhances each agent's observations by providing a best effort understanding of signal propagation across the entire RoI. On the other hand, VDN, leveraging DQN architecture and auxiliary reinforcement learning components, facilitates the coordination of multiple agents for collaborative source localization. In training and inference, all agents utilize the same trained model, but each acts independently based on its observation.

As shown in Figure 1, RoLocMe begins by constructing shared observations that include a building map (updated with all agents' visited positions) and sampled signal points to date from the entire region of interest. SkipNet processes these shared observations to produce a refined signal distribution map. The resulting map is then combined with the shared observations and downsized to form individual observations ϕit for each agent based on its current position. Guided by ϕit , agents use an ϵ-greedy strategy to select joint actions ait and explore new cells. As agents gather new signal samples, the shared observations are updated, allowing SkipNet to refine its predictions and help agents navigate more effectively iteratively. This process repeats until the agents exhaust the search time or when an agent reaches the source.

3.3 SkipNet

SkipNet [Locke et al., 2023] plays a critical role in the success of RoLocMe by leveraging shared observations collected from all visited positions and sampled points across agents. It estimates the signal distribution for the entire RoI. Inspired by the autoencoder architecture proposed in [Teganya and Romero, 2021], SkipNet introduces skip-connections between the encoder and decoder, which resembles a U-Net architecture as shown in Figure 2. These connections minimize information loss during encoding, especially when sampling points are sparse, and enhance gradient flow between the decoder and encoder. This design facilitates efficient and unrestricted information transfer during backpropagation, improving overall estimation.

3.4 Agent Observation Preprocessing

As illustrated in Figure 3, each agent utilizes two key components to construct its observation:

• Shared Observations includes two maps: the Sampling Points Map, which aggregates RSS readings collected over time and scaled between [0, 1], the Building Map, which identifies building pixels (-1), visited cells (1), and unvisited cells (0).
• SkipNet Estimation: A predicted signal propagation map generated by SkipNet, scaled between [0, 1].

Shared observations enhance agents' awareness of each other's positions and provide source estimations based on sampled points. SkipNet generates a best effort signal propagation map for the entire RoI, improving agents' understanding of the signal distribution. An agent i th observation, ϕit , combines shared observations with SkipNet predictions, downsized through bilinear interpolation, and cropped around the agent's position to form a stack of six (d × d) maps. This downsizing and cropping, shown to improve convergence without compromising performance [Alagha et al., 2022], effectively balances global context with local focus.

3.5 Agent Architecture

Figure 4 illustrates the high-level design for our multi-agent system and a detailed view of each agent's neural network architecture. We employ VDN [Sunehag et al., 2017] as a general design for tackling multi-agent cooperatives, Dueling DQN [Wang et al., 2016] for the agent's network design, λreturns [Sutton and Barto, 2018], and [Gallici et al., 2024] suggestions for stabilize DQN in training.

Value Decomposition Network

Value Decomposition Networks (VDN) [Sunehag et al., 2017] provides a straightforward yet powerful approach for coordinating multiple agents. Let θ denotes the network parameters in VDN. VDN assumes the team's optimal Q-value, Qteam,θ , is the sum of each individual agent's Q-value, ˜Qi,θ . Formally:

Agents can then greedily select actions from their local Qvalues to maximize Qteam,θ . During training, the agents share a common reward and update their local Q-values by backpropagating the team's Q-value via Q-learning.

Deep Q-learning

DQN [Mnih, 2013] utilize deep neural network to estimate the action's Q-value directly from raw inputs and update θ according to:

To stabilize learning, DQN employs a large experience replay buffer (for randomized updates) and a target network (updated at a slower pace) to reduce oscillations.

Auxiliary Components

Dueling DQN. Dueling DQN [Wang et al., 2016] separates Q-value estimation in DQN into two distinct streams: the Value stream (V ), which estimates the overall value of a state, and the Advantage stream (Adv), which captures the relative benefit of each action in that state. The Q-value is computed as:

, where st represents the state, at is the current action, at+1 is the action in next state, and Acts is the number of actions. This design mitigates the overestimation of action values and clarifies the state value independently of specific actions.

Parallel Q-learning - PQN. DQN [Mnih, 2013] relies on large replay buffers and a target network for stability but can be slow or prone to divergence without them. Parallel Qlearning [Gallici et al., 2024] tackles this by adding layer normalization to the network and training over multiple vectorized environments in parallel. These changes allow DQN to achieve competitive performance-comparable to policygradient methods like PPO [Schulman et al., 2017]-without large replay buffers and a target network.

λ - returns. λ - returns [Sutton and Barto, 2018] is a target estimation method in reinforcement learning that utilizes bootstrapping, state samples, and rewards. This method weighs short-term and long-term returns by combining n-step returns with the λ parameter, suitable for improving training stability in a partially observable task, as shown in Equation 4. A λ value near 0 favors immediate rewards. In contrast, a value near 1 prioritizes long-term returns.

Reward Design

We investigate the system performance on two reward definitions-dense and sparse- [Alagha et al., 2023]. BreadthFirst Search (BFS) distances between agents and the transmitter are computed at each step.

• Dense rewards: Let Mt be the set of agent-totransmitter distances at time t and min(Mt) their minimum. Let m be the number of agents that moved at time t. Then,

· Sparse rewards:

Termination Conditions

An episode terminates when an agent comes within L meters of the transmitter, marking success or when the maximum step limit is reached, indicating failure.

Training Process

Algorithm 1 details the entire procedure. After each update cycle, multiple validation episodes across diverse settings are conducted, and the best model is saved based on success rates. Before integrating into RoLocMe for agent training, SkipNet undergoes pre-training using mean squared error (MSE) loss.

4 Experiments and Results

This section presents experiments conducted to evaluate the RoLocMe system in comparison with the state-of-arts methods and provides an in-depth analysis of the significance of each component in its design.

Algorithm 1 RoLocMe Training

1: θ ← Initialize network parameters 2: TSC ← 0; t ← 0 ▷ TSC = Training step count 3: SkipNet ← Load the pretrained SkipNet model 4: Shared Obst ←Env Reset ▷ this t is 0 5: SkipNet Predt ← SkipNet(Shared Obst) 6: Fuse Obs ←Concat(SkipNet Predt , Shared Obst) 7: ϕt ← Preprocessing Combine Obs(Fuse Obst) ▷ ϕt = [ϕ0t , . . . , ϕn-1 t ] 8: while TSC ≤ Max Training Steps do 9: for each env e ∈ {0 : Nenv - 1} in parallel do 10: for each agent i ∈ {0 : n - 1} do 11: ait ← πϵ-greedy(ϕit) 12: end for 13: at = [a0t , . . . , an-1 t ] 14: Shared Obst+1, rt, terminalt ← Env Stepe (at) 15: SkipNet Predt+1 ← SkipNet(Shared Obst+1) 16: Fuse Obs ← Concat(SkipNet Predt+1 , Shared Obst+1) 17: ϕt+1 ← Preprocessing Combine Obs(Fuse Obst+1) ▷ ϕt+1 = [ϕ0t+1, . . . , ϕn-1 t+1 ] 18: Store (ϕt, at, ϕt+1, rt, terminalt) in buffer 19: t ← t + 1 20: end for 21: if t mod T = 0 then ▷ Start an update cycle 22: for epoch ∈ {1 : num epochs} do 23: Shuffle buffer by environment trajectories 24: Create mini-batches grouped by environment 25: for each mini-batch - mb do 26: Compute Rλ,mb [t-T:t] using Equation (4) 27: Compute Qteam,θ using Equation (1) 28: Perform gradient descent on 29: ∥StopGrad(Rλ,mb t ) -Qteam,θ (ϕt, at)∥2 30: with respect to the network parameters θ 31: end for 32: end for ▷ End an update cycle 33: Clear buffer 34: end if 35: TSC ← TSC +Nenv 36: end while

4.1 Dataset

We use the RadioMapSeer dataset [Yapar et al., 2022], which consists of 701 environment maps sourced from OpenStreetMap [OpenStreetMap, 2023] and represents various areas across major European cities. Each map features 80 transmitter locations, resulting in 80 distinct signal propagation maps generated using the Dominant Path Model method in the WinProp program [Hoppe et al., 2017]. We divide the dataset by environment, allocating 501 maps for training, 100 for validation, and 100 for testing.

• SkipNet dataset: We created a low-sampling dataset by generating five sampled signals for each propagation map in the training and validation sets using a uniform sampling method with sampling rates between 0.01% and 0.1% of the total pixels.
• RoLocMe dataset: Randomly selected 100 environments each from the training and validation sets, with 5 transmitter placements per environment, totaling 1,000 unique transmissions for end-to-end GPU training. Fi-

nal results use all test maps.

4.2 Methods

We introduce the state-of-art models [Alagha et al., 2023] used for performance comparison:

• MDRL SR-Dense rewards: Multi-agents model trained with Proximal Policy Gradient with dense rewards
• MDRL SR-Sparse rewards: Multi-agents model trained with Proximal Policy Gradient with sparse rewards
• MDRL DC: MDRL SR-Sparse rewards with a guided expert model

To ensure consistency, all comparisons are conducted using a team of four agents and evaluated under the same testing conditions. Each method is trained with 10 different seeds, and performance is recorded across 100 unique starting positions for each proximity zone within every signal propagation map in the testing set. This results in a total of 800,000 starting positions per promimity zone. To achieve a more robust estimate and reduce the impact of outliers, the interquartile mean is used for evaluation [Agarwal et al., 2021]. Table 1 presents the parameters used during the training of RoLocMe.

4.3 Metrics

We introduce three key metrics to evaluate the performance:

• Success rate: Success rate is the percentage of successful localizations, as defined in Equation 7.

Success Rate(%) = #locate transmitter within 5 meters

(7)

#trials

• Localization time: how long a team of agents can successfully localize the transmitter.
• Actions: total number of moves of the agents in the team. The lower the actions, the better the energy efficiency, as each movement costs some energy.

4.4 Performance Comparison with State of Arts Methods

Figure 5 illustrates the convergence rate of each model across various metrics during training. RoLocMe, under both rewarding systems, consistently converges faster than the other models. Among the compared models, MDRL with dense rewards demonstrates slower convergence than RoLocMe.

In Figure 6, as the agents' starting positions move further from the transmitter, the performance of all models degrades. However, RoLocMe consistently outperforms the other models, achieving at least a 30% improvement in the first two proximity groups and approximately 20% in the last two groups. Furthermore, although the number of actions and localization times increases across all methods as proximity increases-due to the longer traversal paths-RoLocMe maintains superior performance compared to the other models.

Additionally, RoLocMe in both rewards functions perform quite similarly. However, there is a slight increase in localizing time and actions taken roughly 3%, which is insignificant. Based on the performance, training RoLocMe with dense reward often yields a slightly better policy than sparse reward. However, using dense or sparse rewards for RoLocMe depends on whether performing repeated BFS calculations for every step during training is feasible.

4.5 Drop-One Component Ablation Study

In Figure 7, RoLocMe's full configuration converges faster and more consistently than its ablation variants (i.e., without λ-returns, layer normalization, dueling, or SkipNet) in both dense and sparse reward settings. These ablated models converge more slowly, especially under sparse rewards, where their early success rates remain near zero for a considerable period. By contrast, RoLocMe trained with dense rewards rapidly achieves near-100% success, requiring fewer actions and shorter localization times.

Figure 8 displays each ablation's performance under both reward functions across various proximity zones. In all cases, RoLocMe maintains higher success rates, shorter localization times, and fewer actions-especially when the agent starts far from the transmitter. While λ- return is also important to the system, removing SkipNet erodes the agents' ability to learn the dynamic of signal propagation, leading to significantly degraded performance. However, the complete RoLocMe design generalizes effectively across all proximities.

Under dense rewards, omitting dueling or layer normalization produces success rates similar to RoLocMe but requires about 2% more time and actions. In this dense reward setting, successes are not distinctly marked and can be obscured by smaller rewards or failures, making λ-returns crucial for properly reinforcing the final objective; without λ-returns, the success rate drops by at least 75%, with localization times tripling and actions increasing by at least 1.5 times compared to the full RoLocMe. Another key component is SkipNet: eliminating it deprives the agent of the ability to learn the dynamic of signal propagation in the RoI, causing success rates to fall by about 15% and time and action costs to rise by roughly 20%.

In the sparse reward setting, models lacking dueling, λreturns, or layer normalization initially achieve comparable success rates within the first three proximity zones; however, the model without λ-returns substantially increases localization time and actions. In the absence of SkipNet, the success rate diminishes, and both localization time and actions increase significantly, reflecting patterns seen in the densereward case. Generally, models missing SkipNet have lower performance and require more time and action to search.

4.6 RoLocMe Scalability on Various Team Sizes

Figure 9 illustrates the performance of the RoLocMe system across various team sizes, ranging from 2 to 16 agents and different proximity groups. For closer proximities (50-100 and 100-150), success rates remain near 100% regardless of team size. However, as proximities increase (150-200 and beyond), larger teams show a decline in performance. While larger teams can cover greater distances more effectively, the increased coordination overhead negatively impacts success rates, leading to higher localization times and more executed actions. We suspect this limitation arises from using VDN, which struggles to optimize policies for large groups of agents.

5 Conclusion

In this paper, we present RoLocMe, a system comprising two deep learning models that complement each other for source localization tasks. By fusing the signal estimation from the SkipNet model into the agents' observations and designing the agents via VDN and DQN, we observed a notable performance boost compared to existing methods. Additionally, we performed an ablation study to illustrate how each component contributes to the agents' performance across various metrics.

Acknowledgments

Research was partially sponsored by the Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-23-2-0014. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.