From General Relation Patterns to Task-Specific Decision-Making in Continual Multi-Agent Coordination

1 Introduction

Collaborative Multi-Agent Reinforcement Learning (MARL) [Oroojlooy and Hajinezhad, 2023] focuses on how multiple agents can coordinate to achieve global goals. Traditional research mainly concentrates on enhancing agents' cooperation abilities in stable environments [Rashid et al., 2020b; Yu et al., 2022]. However, in the real world, agents often face constant changes in diverse tasks [Kiran et al., 2021; Yu et al., 2021; Zhang et al., 2024; Wang et al., 2024a; Wang et al., 2024b]. Particularly, in continual coordination scenarios, agents may lose the ability to perform well in previously learned tasks when adapting to new ones, a phenomenon known as catastrophic forgetting [Robins, 1995] in Continual Learning (CL).

CL seeks to exploit both the shared and the specific properties across different tasks. The properties reflect the essence of continual learning across various domains. For instance, in the class-incremental learning domain, models should understand the overlap or complementarity between new and old classes in feature space, leveraging learned knowledge to adapt to new tasks [Ji et al., 2023]. In the multimodal CL domain, models learn sample-specific and invariant features to construct representations that are broadly adaptable, to prevent catastrophic forgetting [Zhang et al., 2023]. Compared to class-incremental and multimodal CL, the challenges in Continual Multi-Agent Reinforcement Learning(Co-MARL) are more complex due to the uncertain scale of entities. Thus, it is also crucial to figure out the essence of Co-MARL. In this paper, we argue that the essence of continual multi-agent coordination lies in Relation Patterns between tasks.

∗Corresponding author: Kai Lv.

A relation represents the influence of an observable entity on an agent's decision-making. Specifically, the relation between the agent and each entity is determined by the combination of the entity's importance to the agent and the entity's features. In this paper, a relation pattern is formed by integrating the relations of multiple entities. Through experiments, we show that common relation patterns exist across distinct tasks. In Figure 1(a), we apply t-SNE [Van der Maaten and Hinton, 2008] to analyze relation patterns in two cooperative tasks. The red and blue triangles representing the two tasks are tightly clustered, indicating strong similarity in the relation patterns between the two tasks. As the entity scale increases from task 5m vs 6m to task 8m vs 9m in SMAC [Samvelyan et al., 2019], some blue triangles form additional isolated clusters. This reflects that in a more complex observation space, agents can form more relation patterns.

Although the clustered relation patterns exhibit similarity across different tasks, the value of similar relation patterns on decision-making can vary significantly depending on the specific task context. For example, in the 8m vs 9m, agents can win a game by capturing various "focus-fire" patterns (e.g., 4-on-2 or 3-on-2). However, in the 5m vs 6m, the same "focus-fire" patterns may lead to a loss when applying the same policies. This indicates that identical relation patterns result in different returns across tasks. We validate this preference difference in Figure 1(b), where the agents demonstrate noticeable differences in their decision-making preferences (Q-value distribution) across two action spaces.

To capture and exploit these general relation patterns, we propose RPG(General Relation Patterns-Guided Taskspecific Decision-Maker). 1) In capturing the relation patterns, we design a scalable relation capturer that allows agents to measure interactions with other entities from their firstperson perspective, forming general relation patterns. We further add an anti-forgetting regularization to retain key relation patterns across tasks with varying entity numbers. 2) In exploiting the relation patterns, a task-specific decisionmaker is introduced, enabling the agent to respond to dynamic threats and adapt to new collaborative teams. We use a conditioned hypernetwork to generate decision-makers for each task, mapping similar relation patterns to diverse action spaces for flexible adaptation. To ensure the stability of each decision-maker, we add a hypernetwork loss term that prevents forgetting prior cooperation while enabling fast adaptation to new teams. The contributions are as follows:

• We capture the essence of Co-MARL via relation patterns, linking dynamic observation and action spaces.
• We introduce a scalable Relation Capturer that helps agents extract relation patterns, preventing catastrophic forgetting and enhancing cross-task collaboration.
• We inject plasticity into decision-makers through the hypernetwork, enabling flexible decisions based on relation patterns in dynamic environments.
• RPG outperforms existing baselines in continual tasks, excelling on benchmarks like SMAC[Samvelyan et al., 2019] and LBF[Papoudakis et al., 2020].

2 RelatedWork

2.1 Continual Learning in RL andMARL

Continual Reinforcement Learning (CRL) aims to enable agents to learn sequential tasks while avoiding catastrophic forgetting. Early approaches [Rolnick et al., 2019; Chaudhry et al., 2019] alleviate forgetting by reusing past experiences through replay buffers. Regularization-based methods [Kirkpatrick et al., 2017; Aljundi et al., 2018] mitigate forgetting by constraining parameter updates to ensure stable learning. Another class of methods tackles the stability-plasticity tradeoff by dynamically adjusting model architectures, such as employing new networks [Kim et al., 2023] or multi-head structures [Kessler et al., 2022]. However, while these methods have shown progress in single-agent tasks, achieving effective continual learning in multi-agent tasks remains a significant challenge.

In Co-MARL, the complex interactions between agents make continual learning significantly more challenging. Agents must coordinate in diverse teams, requiring them to acquire new knowledge while retaining the ability to address previous tasks. To address these challenges, MALT [Shi et al., 2021] employs progressive neural networks to facilitate knowledge transfer across tasks, while MACPro [Yuan et al., 2024] leverages a shared feature extractor and dynamically expands policy heads to effectively prevent catastrophic forgetting. LL-Hanabi [Nekoei et al., 2021] provides a benchmark for testing continual learning in multi-agent settings. MARR [Yang et al., 2024] adopts a reset strategy to avoid the loss of network plasticity caused by high replay rates. However, these methods fail to thoroughly analyze the fundamental bottlenecks of Co-MARL. In this paper, our approach tackles the core challenges of Co-MARL, providing a stable and efficient solution to address catastrophic forgetting.

2.2 Hypernetworks inMARL

Hypernetworks have demonstrated strong performance in meta-reinforcement learning and continual learning [Beck et al., 2023; Chandra et al., 2023; Ehret et al., 2021]. In MARL, QMIX [Rashid et al., 2020b] introduces hypernetworks into the mixing network to enable effective credit assignment, while MAVEN [Mahajan et al., 2019] is the first to leverage hypernetworks to generate decision-layer parameters, facilitating efficient exploration for agents. HPN [Jianye et al., 2022] employs hypernetworks to achieve permutation equivariance and invariance, enabling it to handle observations with varying dimensions. Concord [Guan et al., 2024] pioneers the use of hypernetworks for continual AI-human collaboration by generating agent parameters to adapt to diverse teammate strategies, though it incurs significant computational overhead and overlooks the challenge of varying agent numbers in continual tasks. In this work, we utilize hypernetworks to generate decision parameters for agents, providing the capability to learn new tasks and achieve stable continual coordination.

3 ProblemFormalization

In a sequence of continual multi-agent coordination tasks, all agents can only access partial local observations. We model this setting as a Continual Decentralized Partially Observable Markov Decision Process (Co-Dec-POMDP) [Oliehoek et al., 2016]. The Co-Dec-POMDP for each task is formalized as a tuple:

where, in task c, Mc denotes the set of all entities in the environment, and Nc ⊆Mc represents the set of agents. At each time step t, the global state st ∈ Sc represents the complete state of the environment. Each agent i ∈ Nc receives a local observation oit ∈ Oc and selects an action ait ∈ Ac. The joint action at = {ait }|Nc| i=1 is then executed, and the environment transitions to the next state st+1 according to the state transition function Pc(st+1 | st, at), while generating a shared reward rt ∈ Rc. The objective of the multi-agent system is to maximize the global value function for the current task: Qtot(s, a) = Est,at [∑∞t=0 γ t r(st, at)], where γ ∈ [0, 1] is the discount factor, and π represents the joint policy.

In a Co-Dec-POMDP, when a new task c + 1 arrives, the entity composition changes fromMc to Mc+1, resulting in corresponding dimensional changes in S, O, and A. Within our continual learning framework, agents are designed to handle these dynamically changing spaces, with the objective of maximizing the global reward in the current task while maintaining the performance of the current agent network across all previously encountered tasks.

4 Method

We believe the core of Co-MARL lies in the relation patterns that manifest distinct properties across different spaces, i.e., observation and action spaces. On the one hand, general relation patterns emerge across tasks with varying numbers of entities, facilitated by the decomposition and combination of the observation space in an entity-wise manner. On the other hand, even similar relation patterns may yield different values when mapped to distinct action spaces. To address this, we propose a General Relation Patterns-guided Task-specific Decision-Maker (RPG) framework. First, to capture and preserve the general agent-to-entity relations from the dynamic observation space, we design a scalable relation capturer with stability. Next, we inject plasticity into the decision-maker using a hypernetwork, enabling it to map the general relation patterns to task-specific action spaces.

4.1 Scalable Relation Capturer with Stability

In sequential Co-MARL tasks, extracting general relation patterns and maintaining their stability is crucial. To this end, we develop a scalable relation capturer to dynamically capture the interaction relationships between entities. The scalable relation capturer has two components: attention-based relation patterns and return-aware anti-forgetting.

Attention-Based Relation Patterns. To extract relation patterns at the entity level, we model the observation with non-fixed dimensions as a combination of multiple entity features. Consider a multi-agent system with m entities. Given the agent's observation ot,i at time step t, we decompose ot,i into the agent's own features ose t,i , n - 1 teammate entity features ote t,i , and m- n other entity features ooe t,i .

Since the relation patterns are independent of the position of entities in the sequence, a permutation-invariant crossattention mechanism can be used to capture general relation patterns. Specifically, as shown in Figure 2(a), we treat ose t,i as the query, while ote t,i or ooe t,i serve as the keys and values, respectively. We then calculate the attention scores for teammate entities and other entities, corresponding to collaborative partners and potential threats or targets, respectively.

where α·t,i represents the attention level of agent i towards teammates and other entities , and √dx is the scaling coefficient. Additionally, we use α·t,i to perform a weighted sum of values, thereby obtaining the relation pattern from the agent's first-person perspective to teammates and others, as shown below:

The relation pattern z·t,i is formed by concatenating the outputs from two cross-attention modules. Furthermore, to establish closer relation patterns with more relevant entities, inspired by MAIC [Yuan et al., 2022] and CoDe [Song et al., 2025], we introduce a regularization term to sparsify the attention scores:

By minimizing the entropy of the attention scores α·i , we encourage the agent to focus on several key entities.

Finally, to capture the agent's information, we encode its own features ose t,i as zse t,i , which is then concatenated with the relation pattern to form zt,i = [zse t,i , zte t,i , zoe t,i ]. It is then fed into the GRU along with the previous hidden state to generate the historical relation pattern ht,i = GRU(zt,i, ht-1,i), which serves as the output of the scalable relation capturer.

Return-Aware Anti-Forgetting. Although we can extract common relation patterns across different tasks, excessive parameter updates of the capturer can lead to the forgetting of past relation patterns. Therefore, applying varying degrees of constraints to the different parameters ensures the stability of the capturer. Inspired by Taylor Pruning [Molchanov et al., 2019], under the i.i.d. assumption of parameters, we measure the importance of a parameter θi by the squared difference in the TD loss when θi is set to zero:

Directly computing the parameter importance using Equation (5) is computationally expensive. Therefore, we approximate the importance of θc j for the c-th task using a Taylor expansion. The second-order Taylor expansion is as follows:

We typically retain only the quadratic term in Equation (6). The final average importance of θc is then given by:

where N denotes the number of samples from the previous buffer, and all parameter importance values of ϕrc are normalized by dividing by the maximum I(θc )max. Specifically, for the importance matrices I(θ1 ), I(θ2 ), . . . , I(θc-1 ) of the past c - 1 tasks, the parameter regularization item can be expressed as the sum of the regularization losses for the relation capturers of each task:

where λrc represents the regularization strength for the relation capturer parameters ϕrc. The learning process can be stabilized by penalizing inconsistencies in the parameters, which helps the agent retain key relation patterns and avoid catastrophic forgetting.

In addition, overly conservative regularization can prevent the agent from extracting all possible relation patterns in more complex tasks. During the early stages of learning a new task, the team's return reflects its ability to solve the current problem. Therefore, we use the team's normalized cumulative return over the first N episodes as the discount factor for Lrc:

where Gn′ norm denotes the raw cumulative return of the n′ -th episode. A larger γp indicates that the current policy is wellsuited to the current task, while a smaller γp suggests relaxing the constraints in the new observation space to learn more potential coordination forms.

4.2 Hyper-Decision-Maker with Plasticity

Another key challenge in Co-MARL is that general relation patterns exhibit varying values across continual tasks. Agents must learn a new decision-making policy to adapt to changing action spaces. Therefore, we introduce a task-conditioned hypernetwork, which aims to generate task-specific decisionmakers that can make rapid behavioral adjustments based on the captured relation patterns.

Using a traditional MLP as the policy layer may lead to a loss of plasticity in the agent. Instead, we utilize a hypernetwork to generate the decision-maker, which further maps the historical relation pattern to the action space. When learning the first task, we use a randomly initialized, Gaussiandistributed, learnable task embedding e1 as the input to the hypernetwork ϕh.

The hypernetwork generates an ego-decision-maker (right) and an interaction-decision-maker (left), as shown in Figure 2(b), corresponding to fixed ego actions and variable interaction actions, respectively. For ego actions, we use the hypernetwork ϕego h to generate the ego-decision-maker f ego ϕh (e1), and then map the historical relation pattern ht,i to a fixeddimension Qt,i(aego|ot,i). For interactive actions, directly mapping ht,i to variable-dimension Qt,i(aint|ot,i) is not feasible. Therefore, we design an entity-wise interaction decisionmaker that focuses on entity interactions. By concatenating the historical relation pattern ht,i with the features of m- n other entities eoe t,i , we map them to each interaction entity's Qt,i(aint(·)|ot,i). Finally, we concatenate the state-action values from both parts to obtain the complete local state-action Qt,i(a|ot,i).

In addition, the hypernetwork itself faces the issue of catastrophic forgetting across tasks. We introduce an additional regularization term for the hypernetworks:

where ˜ ϕh refers to the hypernetworks obtained after the c-1th task. By minimizing the above loss term, we encourage the hyper-decision-maker to learn new behaviors while preserving the decision-maker specific to previous tasks. Furthermore, to ensure a smooth task transition, we initialize the task embedding ec for the new task as follows:

where αinit controls the degree of retention of the previous task's embedding, and gc is the embedding initialized from a Gaussian distribution. This soft initialization method allows the agent to inherit knowledge from previous tasks and smoothly transition to the new task.

4.3 Overall Algorithm

In this section, we provide an overall description of RPG and present the pseudocode of our algorithm in the Appendix.

In the training phase of each task, we train the overall agent network and mixing network using the loss terms in Equation (3) and (4). Additionally, We integrate a multi-head attention module into the mixing network to accommodate the varying state spaces across tasks. This design facilitates the agent's ability to learn collaborative behaviors from a global perspective. A detailed explanation of the mixing network is provided in the Appendix.

Furthermore, during the training phase of the c-th (c > 1) task, to prevent catastrophic forgetting, we introduce regularization loss terms for the relation capturer and decisionmaker. The final overall loss function is as follows:

Lctot = LTD + αattLatt + 1c>1γpLrc + 1c>1Lhy, (12) where αatt is a coefficient used to balance the various loss terms. Notably, 1c>1 is an indicator function that takes the value 0 in the first task (c = 1) and 1 otherwise. Additionally, as tasks are trained sequentially, we evaluate the current agent network on the prior c - 1 tasks to assess the anti-forgetting effect of RPG.

5 Experiments

To validate the effectiveness of RPG, we design comprehensive experiments to address the following questions: 1) Does our method outperform other baselines in learning new knowledge while preventing catastrophic forgetting? 2) How does each component of our approach impact continual learning performance? 3) Does the relation pattern extracted by RPG facilitate better multi-agent coordination compared to other baselines? 4) Does the general relation pattern captured by RPG contribute to zero-shot generalization?

5.1 Experimental Setups

Benchmarks. The experiments in this study are conducted using the Level-Based Foraging (LBF) [Papoudakis et al., 2020] and the StarCraft Multi-Agent Challenge (SMAC) [Samvelyan et al., 2019] benchmarks. The former involves multiple agents collaborating in a 2D grid to collect scattered food items, where the sum of the agents' levels must exceed the food's level to successfully collect it and receive a reward. The latter is a complex real-time strategy game that requires multiple agents to cooperate to achieve objectives, such as destroying enemy units or protecting allied ones. SMAC provides a dynamic state space, heterogeneous unit types, and diverse map settings, effectively evaluating the agents' abilities in both cooperation and competition.

Baselines. To evaluatewhether RGP performswell on these benchmark tasks as they continuously appear, we apply it to a popular value-based method, QMIX [Rashid et al., 2020b]. Additionally, we use MACPro [Yuan et al., 2024], EWC [Kirkpatrick et al., 2017], MAS [Aljundi et al., 2018], Replay, L2, and Finetuning as baselines, where MACPro is the latest method in Co-MARL.

Furthermore, to validate the role of relation patterns in multi-agent cooperation, we combine the relation capturer with QMIX and compare it with common value-based and policy-based methods. Specifically, in SMAC, we use methods such as VDN [Sunehag et al., 2017], QMIX [Rashid et al., 2020b], WQMIX [Rashid et al., 2020a], QTRAN [Son et al., 2019], ResQ [Shen et al., 2022], HPN [Jianye et al., 2022], and FtQMIX [Hu et al., 2023] as baselines. Additionally, we explore policy-based methods like MAA2C, COMA [Foerster et al., 2018], MADDPG [Lowe et al., 2017], MAPPO [Yu et al., 2022], and IPPO [De Witt et al., 2020] in LBF.

5.2 Continual-tasks Performance and Ablation

CL Performance Comparison. In Figure 3, we present a comprehensive result of the continual learning experiments conducted in SMAC, highlighting the superior performance of RPG. RPG outperforms other methods in both mitigating catastrophic forgetting and learning new knowledge. Specifically, the gray background indicates the current task and the training procedure synchronously tests all previous tasks using the current agent parameters. The results show that methods such as EWC, Replay, and MAS experience a performance drop after learning the first task, and fail to achieve optimal performance when learning new tasks, exhibiting weak capabilities in mitigating catastrophic forgetting and acquiring new knowledge. This indicates that directly applying CL methods to MARL settings is not feasible. Finetuning and L2 represent the upper bounds for plasticity and stability, respectively. However, Finetuning exhibits the worst catastrophic forgetting performance as it rapidly forgets knowledge after each task is learned, while L2 regularization, due to its excessive constraints, prevents the agent from adapting to task changes.

The Co-MARL method MACPro, while performing well among the baselines, is inferior to RPG in both single-task performance and plasticity. RPG adapts to 8m vs 9m training faster than MACPro. Moreover, RPG's agent parameter size (112K) is smaller thanMACPro's (377K), andMACPro's dynamic policy head further increases the parameter size. Overall, RPG demonstrates comparable performance to L2 in antforgetting and is more adaptable to new tasks. In addition, we compare RPG with several mentioned baselines on the continual tasks of LBF, as detailed in the Appendix.

Ablation Study. In this section, we investigate the impact of different components in RPG through an ablation study conducted on SMAC. First, we remove the loss term Lrc from the scalable relation capturer to obtain w/o Lrc, in order to study its effect on preventing catastrophic forgetting. Next, we replace Lrc with EWC in RPG (denoted as RPG(EWC)) and apply it to the relation capturer to validate the effectiveness of our parameter importance estimation method. Then, in w/o Lhy, we eliminate the hypernetwork regularization loss term Lhy to investigate whether a free hypernetwork could still facilitate continual learning. Additionally, we introduce RPG(MLP), which replaces the decision-maker generated by the hypernetwork with an MLP for decision-making. Finally, we remove both Lrc and Lhy from RPG, and we refer to this setup as w/o Lrc, Lhy.

Continual learning aims to balance stability and plasticity, requiring joint analysis of Table 1 and Figure 4 for a comprehensive assessment. In Figure 4, we test the algorithm on 5m vs 6m at four stages during training to evaluate its forgetting resistance. Table 1 shows the algorithm's performance at different stages of task completion. We find that w/o Lrc, lacking regularization, improves plasticity, but its performance rapidly degrades when the second task begins. This occurs because relaxing the relation capture constraints enhances the agent's learning capacity but compromises its memory retention. Similarly, RPG(EWC) shows a similar performance drop, confirming the reliability of our parameter importance estimation. Removing Lhy allows w/o Lhy to continue learning new tasks, but its forgetting resistance declines, highlighting the need for a constrained hypernetwork. Notably, while RPG(MLP) maintains coordination on 5m vs 6m, it loses some plasticity, emphasizing the necessity of task-specific decision-makers. This is because the MLP cannot isolate the decision-makers of different tasks. Finally, w/o Lrc, Lhy loses the ability to resist forgetting.

5.3 Single-task Performance

In this section, we conduct comparative experiments to validate the coordination performance of our method in single tasks. Figure 5 and 6 show the comparison of RPG with baselines in SMAC and LBF.

In several super-hard maps of SMAC, RPG achieves the best performance, demonstrating that agents dynamically selecting interaction relationships with different teammates or opponents based on their own states is more conducive to achieving better coordination. In the LBF, particularly in maps with more entities and larger areas, RPG outperforms other baselines. This advantage is attributed to the ability of the relation pattern capturer to quickly identify key interactive entities in large-scale tasks, enabling efficient collaboration. These results indicate that the relation pattern is not only effective in complex, continual multi-agent cooperation tasks but also exhibits superior performance in more challenging single-task scenarios.

As shown in Figure 7, we conduct an ablation study on the sparse attention mechanism using w/o Latt to analyze its impact on the generation of relational patterns by agents. The results indicate that removing sparse attention slightly reduces the performance of RPG, confirming that interacting with a few key entities facilitates better collaboration among agents.

5.4 Zero-shot Performance

In addition to learning sequential tasks, agents should also learn to leverage prior coordination experience to handle unseen tasks. To end this, we compare RPG with EWC and MAS to assess its zero-shot generalization capability in tackling novel tasks. Specifically, in the SMAC scenarios, agents trained on continual tasks {5m vs 6m, 12m, 5m, 8m vs 9m} are tested on 5 additional unseen tasks to validate the generalization performance. As shown in Table 2, compared to other baselines, RPG demonstrates superior generalization performance in 4 out of the 5 tasks. However, RPG's generalization ability on 3m does not match that of MAS. This indicates that in simple tasks with few agents, complex relation patterns may instead hinder simple cooperation. This highlights the ability of RPG to extract and retain generalizable relation patterns, enabling robust generalization to new tasks.

6 Conclusion

This paper focuses on the problem of continual coordination in MARL. To address this issue, we analyze the essence of continual multi-agent coordination-relation patterns-and propose a novel framework, RPG. RPG extracts general relation patterns from dynamic observation spaces and maps them to varying action spaces. Results demonstrate that RPG consistently outperforms baselines across various scenarios. However, RPG focuses solely on homogeneous multi-agent collaborative tasks and currently lacks the capability for perpetual knowledge acquisition. In future work, we aim to address these challenges. We hope RPG inspires further interest in continual multi-agent coordination research.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities (Grant No. 2024XKRC080) and the National Key Laboratory of Air-based Information Perception and Fusion (Grant No. 202300010M5001).