Guiding Team Objectives with Individual Policies: A Two-Stage Model Aggregation Framework for Partially Cooperative MARL

1 Introduction

Multi-agent reinforcement learning (MARL) has succeeded in fully cooperative scenarios [Liu et al., 2025; Li et al., 2022]. However, in real-world settings with partial cooperation, where agents must balance individual and team objectives, such as football players striving to win as a team while showcasing individual performance, MARL cooperation can be undermined by conflicts between these objectives.

∗Corresponding author.

Theoretically, using team rewards exclusively to train MARL agents [Foerster et al., 2018; Rashid et al., 2020] can enable cooperation in Partially Cooperative Markov Games (PCMG). However, this fully cooperative setting faces endogenous challenges in credit assignment and exploration, especially when team rewards are sparse [Wang et al., 2022; Xu et al., 2023; Li et al., 2024], making it nearly infeasible to train a cooperative policy. To mitigate these issues, the effective use of individual rewards to guide the training of cooperative policies emerges as a crucial and promising direction in partially cooperative settings. Early works [Durugkar et al., 2020; Ma et al., 2024; Chen et al., 2022] have explored combining individual and team rewards to directly incentivize cooperation, defining the PCMG as an independent learning game (Fig. 1(a)). However, combining multiple rewards often necessitates a detailed understanding of the specific research scenario, limiting the generalizability of such methods. As a result, recent effort [Wang et al., 2022] has sought to combine the strengths of individual and team policies through policy distillation (PD) [Rusu et al., 2016; Lai et al., 2020], as illustrated in Fig. 1(b), to facilitate training of team policies based on individual policies or vice versa. However, PD is data driven and suffers from incomplete knowledge transfer, as it is constrained by its inability to capture policy information outside the distribution of the distillation data, limiting the efficiency of policy mutual learning. These limitations highlight the need for more direct, data-independent methods.

Unlike PD, model aggregation (MA) [Bourdais and Owhadi, 2025; Li et al., 2025], offers a direct and effective approach to integrating multiple policies, and has gained widespread application in fields such as federated learning [Wang et al., 2020; Lee et al., 2023] and ensemble learning [Chan and van der Schaar, 2022; Kotary et al., 2023]. In reinforcement learning (RL), a notable example is A3C [Mnih et al., 2016], which trains multiple policy models in parallel and uses MA to improve training efficiency and policy generalization. However, MA still suffers from instability, mainly due to the volatility of individual policy models as aggregation objects and the discontinuous parameter updates during aggregation, both of which can cause significant performance fluctuations or even severe degradation.

To this end, this paper introduces Two-Stage Model Aggregation (TSMA), as depicted in Fig. 1(c), a multi-policy MA framework for MARL. TSMA leverages individual policies to guide optimal team policy learning in partially cooperative scenarios. It achieves objective consistency through model aggregation, avoids the data dependency of knowledge distillation, and improves learning efficiency by integrating intermediate-layer knowledge. TSMA consists of two adaptive aggregation stages. In the first stage, each agent aggregates two individual policies trained on the same individual reward using loss-based adaptive weights to enhance expressiveness and stability. In the second stage, the aggregated individual policy is further combined with the team policy trained on team rewards, where the aggregation weights gradually decrease over training to enable smooth knowledge transfer and effective team learning. To enhance individual policy diversity, TSMA introduces parameter-level variation in individual policies after aggregation, providing a broader range of policy options for subsequent training. Additionally, TSMA applies L2 regularization across all policy models to mitigate instability caused by aggregation-induced abrupt parameter updates. The TSMA agent interacts with the environment via the team policy, making it a natural extension compatible with all Actor-Critic (AC)-based MARL algorithms. To demonstrate its generality, we integrate TSMA into two representative AC algorithms: MAPPO [Yu et al., 2022] and MADDPG [Lowe et al., 2017]. These methods are selected because they exemplify the on-policy and off-policy learning paradigms, respectively, providing complementary characteristics in terms of sample efficiency and training stability. We then evaluated TSMA-MAPPO and TSMA-MADDPG on PCMG benchmarks, including the Multi-Agent Particle Environment (MPE) [Mordatch and Abbeel, 2018] and the Pursuit scenario of the Stanford Intelligent Systems Laboratory (SISL) [Gupta et al., 2017] at PettingZoo. The results show that TSMA-MAPPO and TSMA-MADDPG outperform the state-of-the-art methods in cooperative performance within PCMG while ensuring stable aggregation.

The contributions of this paper are summarized as follows: (1) We introduce TSMA, a multipolicy MA framework for MARL, designed to address cooperation challenges in partially cooperative scenarios by guiding team policy learning through individual policies. (2) TSMA is a versatile aggregation framework compatible with all AC-based MARL algorithms. We demonstrate its effectiveness by integrating it into both on-policy (MAPPO) and off-policy (MADDPG) based algorithms. (3)We provide a theoretical analysis of the convergence of TSMA, and through extensive experiments in partially cooperative scenarios, confirm its higher cooperative efficiency over state-of-the-art baselines.

The paper is structured as follows. Sect. 2 reviews related work. Sect. 3 introduces the PCMG model. Sect. 4 details the TSMA framework and algorithm with qualitative analysis. Sect. 5 presents experimental results and analysis. Finally, Sect. 6 provides the conclusion and future work.

2 RelatedWork

Mixed-reward cooperation methods. The mixed-reward method incorporates individual rewards alongside team rewards, enabling it to train policies using a combination of both in PCMG. For instance, Durugkar [Durugkar et al., 2020] proposed a weighted combination of individual rewards and team rewards to promote effective learning of cooperative policies. Some works directly incentivize cooperative behavior through tailored rewards. ReLara [Ma et al., 2024] introduces an auxiliary reward agent to automatically learn dense reward functions, allowing the principal policy agent to benefit from auxiliary and sparse team rewards. Similarly, DCIR [Lin et al., 2023] rewards or penalizes agents based on their behavioral consistency with others and employs a dynamic scale network to dynamically adjust reward strength, allowing agents to flexibly modify their level of cooperation based on environmental changes. Mixing multiple rewards often relies heavily on expert experience to fine-tune the weighting coefficients of multiple rewards. Additionally, dimensional inconsistencies among rewards can complicate direct weighting. Furthermore, the independent reinforcement learning-based training paradigm used in these methods may struggle with the non-stationarity problem.

Policy mutual learning methods. Mutual learning methods for policies focus primarily on designing loss functions that ensure consistency between individual and team goals, quantifying policy differences, and utilizing these to guide policy optimization or achieve goal consistency between policies through shared loss functions. For example, IRAT [Wang et al., 2022] employs knowledge distillation to efficiently align individual and team policies by applying differentiated constraints. Similarly, TRP [Wang et al., 2024] predicts truncated returns of historical trajectories and minimizes the prediction error between current and historical policies, thus achieving consistency in the value function between different policies. D3C [Kwon et al., 2023], on the other hand, adjusts each agent's loss function by learning a set of shared weights that combine individual losses into a modified objective, encouraging alignment with the global objective. Although data-driven methods effectively ensure that cooperative policies remain consistent with global goals, they are inherently limited in extracting policy information beyond the data distribution. This limitation restricts their ability to achieve more efficient policy mutual learning.

Model aggregation methods. Model aggregation, a core technique in federated learning [Wang et al., 2020; Karimireddy et al., 2020], combines multiple client models trained on their respective local datasets to produce a more accurate and generalized global model while preserving data privacy and security. This technique has shown effectiveness in RL, e.g., A3C [Mnih et al., 2016], which improves learning efficiency by training multiple agents in parallel environments and aggregating their models. Beyond efficiency, MA is crucial in achieving objective consistency. AgA [Li et al., 2025] aligns individual and team objectives by performing a weighted aggregation of gradients from different policies to generate a global gradient. Similarly, SPAHM [Yurochkin et al., 2019] maps local model parameters to a unified parameter space through parameter matching and leverages MA to minimize parameter distribution differences, ensuring stability during learning. While some of these methods address the stability issues in aggregation, they often suffer from high training complexity. Additionally, they fail to account for the prioritization of multiple models, which is critical in PCMG scenarios, as policies trained for different objectives may conflict. In this work, we focus on leveraging individual policies to guide team policy learning while maintaining stability throughout the aggregation process.

3 Preliminaries

The Partially Cooperative Markov Game (PCMG) can be formalized as a seven-tuple: < N, S,A,P,O, γ,R >, where N = {1, ..., n} is the set of agents, S is the global state space of the environment and A is the action space for each agent. The state transition function P describes the probability distribution of the environment changing from state s to next state s′ given the joint action at = (a1,t, ..., an,t), ∀ai ∈ A. O represents the partially observable state space and γ is the discount factor. R is the reward function that comprises each agent's individual reward and the shared team reward based on the current state and the joint action. At the time step t, all agents execute a joint action at. The environment provides each agent with two rewards for at, i.e., individual reward ridv i,t and shared team reward rteam t .

In PCMG, each agent has two objectives, i.e., individual and team objectives. The individual objective is to maximize their discounted cumulative individual return, J idv i (π) =

E[∑∞t=0 γ t ridv i,t ]. All agents cooperate to learn a joint policy simultaneously, to maximize the cumulative discount return for the entire team, J team i (π) = E [∑∞t=0 γ t rteam t ]. In PCMG, agents pursue individual objectives while also cooperating with others to achieve team objectives. However, discrepancies in reward functions among agents create only partial alignment of individual rewards, often leading to incomplete cooperation. This misalignment can cause the team objectives of the agents to converge to the local optimum rather than the global optimum. In this paper, we address the challenge of balancing the conflict between individual and team objectives in PCMG, aiming to maximize team benefit.

4 Method

To address the challenges of PCMG, we propose Two-Stage Model Aggregation (TSMA), a multi-policy MA framework for MARL that uses individual policies to guide the learning of optimal team policy. Fig. 2 shows the learning process of a TSMA agent, including data-based training and a twostage model aggregation. Within this framework, each agent is equipped with a team actor for interaction with the environment and two individual actors to guide the team policy. Individual and team critics separately evaluate individual and team returns, guiding the learning of the actors. TSMA, as a universal framework, seamlessly integrates with all standard AC algorithms. In this paper, we implement TSMA under the CTDE architecture by integrating it with two representative MARL backbones: the on-policy MAPPO and the off-policy MADDPG, forming TSMA-MAPPO and TSMA-MADDPG respectively. This section first details the TSMA aggregation process, followed by the training details of TSMA-MARL and the convergence analysis of the TSMA.

4.1 TSMA

In this section, we detail TSMA, which focuses on enabling individual policies to promote effective learning of team policies through MA. The core challenge is addressing MA's instability, which arises from two main sources: the inherent volatility of individual policy models and the non-asymptotic nature of aggregation updates. Specifically, TSMA first performs a weighted aggregation of two individual policies (trained on the same individual reward). It then integrates the aggregated individual policy with the team policy (trained in team reward) using a structure-level MA approach, bypassing data dependency in conventional policy distillation. To further enhance individual policy diversity, TSMA introduces parameter-level variation. Additionally, TSMA incorporates L2 regularization to guide all policy parameters toward a consistent norm space, mitigating the instability caused by the non-asymptotic nature of the aggregation updates. Note that reference [Dohare et al., 2024] provides precedents for the use of variation injection and L2 regularization, demonstrating their theoretical foundation and effectiveness.

Aggregation of individual policies: Model aggregation can efficiently transfer knowledge from individual to team policies, but the instability of individual policies can lead to fluctuations or degradation in team policy performance. This issue arises because these individual policies, as foundational guidance signals, directly influence team policy learning outcomes. To mitigate this problem, we train two individual policies for each agent and use their aggregated results to guide the team policy, enhancing the expressiveness and stability of individual policies. Furthermore, to reduce the interference that noise from low-quality policies might cause in the aggregation results, we employ a weighted aggregation that dynamically assigns weights based on the loss of each individual policy, highlighting the contributions of high-quality policies and suppressing the interference from low-quality ones. This 'majority voting, retaining the best' ensemble strategy effectively improves the stability of policy aggregation.

where ¯w = exp(-L1)/(exp(-L1) + exp(-L2)) signifies the aggregated weight between individual policies and Lj denotes the average loss value of the individual policy θidv,j based on data-driven training.

Aggregation of individual and team policies: The core of this study is to integrate individual and team policies for each agent to help learn the more challenging team policy, ultimately improving the efficiency of multi-agent cooperation in PCMG. The approach of using individual policies to guide the learning of team policies has been proven to be effective in IRAT [Wang et al., 2022], but the policy distillation method used in IRAT is limited by data distribution and struggles to achieve comprehensive policy transfer. To overcome this limitation, we propose replacing the data-driven distillation method with model aggregation. Model aggregation not only offers higher efficiency, but also bypasses the dependence on data. Integrates individual and team policies at the structural level, thus supporting the learning of more complex cooperative policies. To further improve the stability of the aggregation process, we enable individual and team policies to leverage their respective strengths at different training stages. Specifically, we introduce a weight annealing function ˆw(e) from MAPTF [Yang et al., 2021] to adjust the aggregation ratio between individual and team policies, ensuring that the importance of the team policy progressively increases throughout the training process. This design retains the rich exploratory signals provided by individual policies during the early stages of training, while also ensuring that team policies can effectively lead to cooperative behaviors in later stages. The process of aggregating individual and team policies can be formalized as:

Variation injection: Although different individual policies can effectively help to learn team policy models and help escape local optima, applying the aggregated individual and team policy model θteam directly to all individual policies can lead to homogenization. To prevent this, we have introduced a variation mechanism to ensure the diversity of individual policies. Specifically, the aggregated model is updated through two paths: one directly updates the team policy, and the other updates the individual policies after variation injection. To enhance policy diversity while maintaining model stability, the TSMA performs fine-grained variation control at the parameter level. For individual policy models with the same structure, TSMA adjusts the variation magnitude based on the variance of corresponding parameters in each agent's individual policies. Parameters with higher variance, indicating greater behavioral differences between individual policies, will undergo larger variations to explore a wider solution space. In contrast, parameters with lower variance, which exhibit stability across different policies, will receive smaller variations to maintain model stability. Ultimately, the parameters of each individual policy model follow the distribution:

L2 regularization: The non-asymptotic nature of aggregation updates means that significant discrepancies between the parameter distributions of the policies to be aggregated can cause the resulting parameters to deviate from the original distribution characteristics when directly applying aggregation. Such deviations may cause the parameters to fall into ineffective regions of the parameter space, ultimately causing a sharp decline in policy performance[Dohare et al., 2024]. This phenomenon fundamentally arises from the lack of a unified metric in the parameter space, where different policies may reside in entirely different scales or distributional forms. To effectively address this issue, we apply L2 regularization to force all aggregated policy parameters to contract toward a common canonical space. This alleviates inconsistencies and ensures that the aggregated policy retains the core features of the original. We incorporate L2 regularization penalties into the loss functions of both individual and team policies.

4.2 Implement TSMA intoMARL

As mentioned above, TSMA is a universal framework compatible with all standard AC algorithms. To comprehensively evaluate its compatibility across different learning paradigms, this paper selects MAPPO and MADDPG as backbones, which are representative on-policy and off-policy methods, respectively, producing the TSMA-MAPPO and TSMA-MADDPG algorithms. Within the CTDE framework, both TSMA-MAPPO and TSMA-MADDPG employ individual critics {φidv i }i∈N and a team critic φteam to estimate individual and team objectives based on individual rewards {ridv i }i∈N and the shared team reward rteam. In practice, we share the parameters of the individual critics across all agents to enhance training efficiency. Each agent is equipped with three actors, including two individual actors parameterized by {θidv,1 i , θidv,2 i } and a team actor θteam i , where the team actor is responsible for interacting with the environment and sampling data. The individual and team actors' policies are evaluated by their corresponding critics. In TSMA-MADDPG, both critics and actors are additionally associated with their respective target networks to stabilize training.

The TSMA training process for MARL algorithms consists of two stages: the data-based training stage and the model aggregation stage, as described in Sect. 4.1. During the databased training phase, the individual and team critics, together with the actors of each agent, are updated following the standard training paradigms of the underlying algorithms (e.g., MAPPO or MADDPG), utilizing trajectory data collected through the interaction of the team policies with the environment. As detailed in Sect. 4.1, to ensure stability during the model aggregation stage, L2 regularization is applied to the loss functions of individual and team actors. This regularization effectively constrains the parameters of the aggregation model within a consistent distribution space. The actor's loss function for each agent is defined as follows:

where θi ∈ {θidv,1 i , θidv,2 i , θteam i }, i ∈ N, λ is the hyperparameter that controls regularization intensity, and ∥θ∥22 is the L2 norm square of the parameter.

4.3 Convergence Analysis

This study adopts the FedAvg convergence analysis framework [Li et al., 2020], as both methods share the fundamental mechanics of multiple local updates, then central aggregation.

Let J(θ) denote the objective function, and assume it is Lsmooth. Under stochastic gradient descent with step size ηt, the expected improvement per iteration satisfies

where gt is the stochastic gradient. The term E∥gt - ∇J(θt)∥2 captures the variance due to client sampling and local drift. By invoking bounded variance assumptions and gradient deviation bounds from FedAvg, it can be shown that

Substituting this bound leads to

Since J(θ) is bounded below, E[J(θ0) - J(θT )] is finite. Therefore, for a fixed learning rate η:

implying convergence to an O(η)-neighborhood of a stationary point. By decaying η appropriately, the algorithm can asymptotically approach an exact stationary point.

5 Experiment

TSMA is a general framework applicable to different MARL algorithms, as evidenced by its integration with both MAPPO and MADDPG in the main comparative experiments. For ablation and sensitivity studies, we use MAPPO as the representative backbone to focus on analyzing TSMA's internal mechanisms. In this section, we conduct experiments to answer the following questions: RQ1: Can TSMA-MARL enable efficient teamwork in partial cooperation scenarios? RQ2: How does the degree of variation injection affect the performance of TSMA-MAPPO? RQ3: How do different components (i.e., L2 regularization, variation injection, multiple individual policies, and weight annealing function) affect TSMA? RQ4: How do individual policy counts and coefficient weighting of variation affect TSMA? RQ5: How does TSMA-MARL perform as the number of agents scales up?

5.1 Experiment Settings

This section will introduce the experiment's scenario setting and comparison methods.

Scenario Setting: To evaluate the team cooperation capability of TSMA-MAPPO and TSMA-MADDPG in partially cooperative scenarios, we conduct experiments across four challenging environments: Predator-Prey, Spread, and Attack from the MPE, and the Pursuit task from the SISL suite. These environments feature a hybrid reward structure, requiring agents to balance individual gains (e.g., distance-based rewards) with sparse, high-value team rewards that are only triggered upon successful team coordination.

Compared Methods: We evaluated TSMA-MAPPO and TSMA-MADDPG against six RL baselines grouped by reward design: independent learning with individual rewards (IPPO [Schulman et al., 2017]), cooperative learning with team rewards (MAPPO [Yu et al., 2022], HAPPO [Kuba et al., 2022]), and hybrid approaches combining both (RS [Durugkar et al., 2020], IRAT [Wang et al., 2022], AgA [Li et al., 2025]). To fairly compare the algorithms, we use the same network architecture and hyperparameters (summarized in Tab. 1). Notably, Tth = 0 is optimal in the Pursuit scenario due to individual-team policy conflicts, and this value is used in all experiments except the comparative study.

5.2 Experiment Results

Comparison with SOTA methods (RQ1): Fig. 3 shows that TSMA-MAPPO and TSMA-MADDPG achieve the best cooperative performance across all four partially cooperative tasks, with TSMA-MADDPG exhibiting superior data efficiency in 'Predator-Prey' and 'Attack'. Performance swings

(b) Hyperparameter (c and β) comparison

Figure 5: Performance comparisons of TSMA-MAPPO ablations and hyperparameter studies in Predator-Prey, averaged over the last 200 episodes across five seeds.

of TSMA-MAPPO in Predator-Prey, possibly related to variation, prompted further experiments in RQ2. Among baselines, IRAT yields suboptimal cooperation as data limitations restrict individual-team policy mutual learning. IPPO, RS, and AgA deliver strong individual performance but fail in team cooperation: IPPO is limited by individual rewards, RS suffers from fixed reward weighting that causes optimization interference, and AgA's fixed-ratio gradient aggregation lacks task-driven adaptation, hindering effective collective behavior despite strong individual performance. MAPPO and HAPPO fail to learn cooperation due to sparse team rewards.

Impact analysis of variation injection degree (RQ2): We hypothesize that TSMA-MAPPO's performance fluctuations in Predator-Prey are caused by variation injection. To verify this, we analyze the correlation between the degree of variation (measured by individual model parameter variance) and algorithm performance. Fig. 4 shows a clear positive correlation between performance fluctuations and parameter variation magnitude after the initial training phase (dashed line). This confirms variation's key role: insufficient variation limits diversity and traps models in suboptimal solutions, while moderate variation aids in escaping local optima.

Ablation study (RQ3): We conduct ablation studies on TSMA components in the Predator-Prey scenario (Fig. 5(a)). We analyze four variants of TSMA-MAPPO: removing L2 regularization (-L2), variation injection (-V), the multiple individual policies(-MP), or weight annealing (-WD). The results confirm each component's critical role. Both -V and -MP suffer from insufficient policy diversity, leading to suboptimal cooperation. The absence of L2 regularization causes parameter divergence and model collapse, while fixed weight aggregation in -WD impedes the dynamic knowledge transfer required across training, hindering team policy learning.

Hyperparameter analysis study (RQ4): We present a sensitivity analysis of two TSMA hyperparameters, the number of individual policies c and the variation intensity β, using the Predator-Prey scenario (Fig. 5(b)). The left panel reveals that maintaining two or three individual policies yields optimal performance. A single policy provides insufficient information for optimizing team policies, while too many policies degrade performance due to inefficiencies; thus, we set c = 2 as the default. The right panel indicates that the optimal variation weight is β = 0.0001, as lower values have negligible effect while higher values disrupt cooperation.

Scalability study (RQ5): To verify scalability, we test our methods on an extended Pursuit-large environment with 30 pursuers, 50 evaders, a 40x40 map, and a 100-step episode limit. As shown in Fig. 6, TSMA-MADDPG and TSMAMAPPO maintain optimal performance at scale. TSMAMADDPG's data efficiency becomes more advantageous with more evaders. Both methods achieve the best team cooperation despite lower individual rewards, showing robustness to individual-team objective conflicts. IRAT performs poorly due to limited exploration under short episodes, while AgA is omitted due to prohibitive computational cost at this scale.

6 Conclusion

To promote cooperation in partial cooperative multiagent scenarios, we introduce TSMA, a multipolicy model aggregation framework for MARL. TSMA aligns individual and team objectives, allowing individual policies to actively support team policy optimization and promoting efficient cooperation. As a versatile framework compatible with standard AC algorithms, we implement it with MAPPO and MADDPG to cover both on-policy and off-policy paradigms, thus developing TSMAMAPPO and TSMA-MADDPG. Experimental results in various scenarios in the MPE and the SISL Pursuit scenario demonstrate that TSMA-MARL significantly enhances team cooperation in partial cooperative scenarios.

Despite notable progress in enhancing algorithm diversity via variation and multi-policy training, the method still suffers from training fluctuation and high computational cost, leaving room for further optimization. Future work will refine the variation mechanism to improve stability while preserving policy diversity. Another promising direction is to adaptively adjust the number of individual policy models, so as to efficiently utilize limited policies, lower training costs, and build a more effective MARL framework.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported by the National Natural Science Foundation of China with Grant No. 62376048.