CADP: Towards Better Centralized Learning for Decentralized Execution in MARL

1 Introduction

Cooperative Multi-Agent Reinforcement Learning (MARL) has recently been attracting increasing attention from research communities, attributed to its capability on training autonomous agents to solve many real-world tasks, such as video games [Vinyals et al., 2019], traffic light systems [Wu et al., 2020] and smart grid control [Xu et al., 2024]. However, learning cooperative policies for various complex multiagent systems remains a major challenge. Firstly, the joint action-observation space grows exponentially with the number of agents, leading to a scalability problem when considering the multi-agent system as a single-agent one to search the optimal joint policy [Rashid et al., 2018; Sunehag et al., 2018]. Moreover, optimizing agent policies individually also suffers from non-stationarity due to the partial observability constraint [Hong et al., 2022; Jiang et al., 2024]. To tackle these problems, Centralized Training with Decentralized Execution (CTDE) is proposed as a popular learning framework for MARL [Lowe et al., 2017]. In CTDE, as depicted in Figure 1(a), decentralized agent policies are trained by a centralized module with additional global state information, while agents select actions only based on their own local observation without any communication

In recent years, the CTDE framework has been widely used in MARL, including Value Decomposition (VD) methods [Sunehag et al., 2018; Rashid et al., 2018; Wang et al., 2021; Son et al., 2019; Rashid et al., 2020; Liu et al., 2023; Liu et al., 2024; Kapoor et al., 2024] and Policy Gradient (PG) methods [Lowe et al., 2017; Foerster et al., 2018; Yu et al., 2022; Kuba et al., 2022], which achieves the stateof-the-art performance in different benchmarks. Despite its promising success, we argue that the centralized training in CTDE is not centralized enough. This is to say, the existing CTDE framework cannot take full advantage of global information for centralized training. Specifically, agent policies are assumed to be independent of each other [Wang et al., 2023], and the existing CTDE framework only introduces global information in the centralized module, while agents are not granted access to global information even when centralized training. This partial observability limits agents to search better global joint policy [Hong et al., 2022;

Chen et al., 2024; Zhao et al., 2022].

To remedy this issue, several prior efforts propose to design a teacher-student CTDE framework [Hong et al., 2022; Chen et al., 2024; Zhao et al., 2022], as depicted in Figure 1(b). These works enable teacher agents to use the global state information during centralized training, while student agents with local observation can imitate the behaviors from teacher agents via knowledge distillation. However, these works just simply take the additional state information as the input of agent policy, which follows the independence assumption on agent policies. Therefore, the agents still make their own decisions without considering the policies of other agents during centralized training. In this way, the expressiveness of the joint policy is inevitably limited, leading to an inefficient joint exploration and perception, which can degrade the final performance.

In this paper, we propose a novel Centralized Advising and Decentralized Pruning framework, termed as CADP, to enhance basic CTDE with global cooperative information. As depicted in Figure 1(c), CADP enables agents to exchange advice with each other instead of only using global state information during centralized training. This approach is aligned with a common way of human communication, where humans often offer helpful and tailored advice based on their knowledge and beliefs instead of simply providing their own information [Stenning et al., 2006]. Therefore, the centralized advising mechanism allows agents to deliver teamoriented actions for better cooperation. To generate the final decentralized policies, we further propose to smoothly prune the dependence relationship among agents via a dedicated auxiliary loss function.

Our main contribution is the dedicated attempt that adopts agent communication to enhance basic CTDE framework for fully centralized training with decentralized execution. We propose a novel Centralized Advising and Decentralized Pruning (CADP) framework to promote explicit agent cooperation during training while still ensure the independent policies for execution. CADP is designed to provide a new general training framework for different MARL methods based on CTDE. Experiments conducted on various benchmarks show that the proposed CADP framework yields results superior to the state-of-the-art methods.

2 RelatedWorks

Teacher-student CTDE Framework. Conventional CTDEbased methods fall short of fully utilizing global information for training, leading to an inefficient exploration and perception of the joint-policy and resulting in performance degration. Therefore, several recent works attempt to improve CTDE with the teacher-student framework. IGM-DA [Hong et al., 2022] firstly proposes centralized teacher with decentralized student frameworks, where the teacher models use the global state while the student models use the partial observation. It adds an additional knowledge distillation loss to enable the teacher model to assist in training the student model. In CTDS [Zhao et al., 2022], the teacher model still uses the agent observation, but the field of view is set to infinite during centralized training. PTDE [Chen et al., 2024] trains a network to aggregate local observation and global state, resulting in a better representation of agent-specific global information. Most of these methods focus on distillation to transfer the knowledge from the teacher to the student for decentralized execution, which is essentially an offline imitation learning. Thus, the agents still ignore the policies of other agents and make their own decision during centralized training, resulting in an inefficient cooperative exploration.

Communication in MARL. Communication [Wang et al., 2020a; Sukhbaatar et al., 2016; Li et al., 2023; Nayak et al., 2023; Lo et al., 2024] is also widely used in MARL, which facilitates information transmission between agents for effective collaboration when agents can only access their own observations. Recently, many methods have used attention as the main mechanism of communication [Das et al., 2019; Jiang and Lu, 2018; Iqbal and Sha, 2019; Yuan et al., 2022; Mao et al., 2020b; Hu et al., 2024], where self-attention can be considered as a message exchanging mechanism with other agents. The communication paradigm allows agents to communicate during both the training and execution stages. In contrast, CTDE emphasizes centralized training of agents while they execute their individual policies in a decentralized no-communication manner. Furthermore, to address communication constraints, various message pruning methods [Wang et al., 2020b; Mao et al., 2020a; Wang et al., 2020c; Yuan et al., 2022; Ding et al., 2020] propose to compress and refine communication information, choose with whom to communicate. Furthermore, MACPF [Wang et al., 2023] offers a method involving information transmission during training without necessitating it during execution. However, it only enables unidirectional forwarding of previous agents' actions, which does not qualify as comprehensive mutual communication.

3 Preliminary

3.1 Dec-POMDP

We consider a fully cooperative multi-agent task as the Decentralized Partially Observable Markov Decision Process (Dec-POMDP), which is defined as a tuple ⟨A, S, U, P, r,Ω,O, γ⟩, where A = {an}Nn=1 is the set of N agents and s ∈ S is the global state of the environment. At each time step t, each agent an ∈ A receives an individual partial observation ont ∈ Ω drawn from the observation function O(st, an), then each agent chooses an action unt ∈ U which forms joint action ut ∈ U N . This causes a transition to the next state st+1 according to the state transition function P(st+1|st,ut) : S × U N × S → [0, 1]. The reward function which is modeled as r(st, ut) : S × U N → R and γ ∈ [0, 1) is the discount factor. Each agent an has an actionobservation history τ n ∈ T ≡ (Ω × U)∗ , on which it conditions a stochastic policy πn (un |τ n ) : T × U → [0, 1]. The joint action-observation history is defined as τ ∈ T N . In this work, the joint policy π is based on joint action-value function Qtot t (st,ut) = Est+1:∞,ut+1:∞ [∑∞i=0 γ i rt+i | st,ut ] . The final goal is to get the optimal policy π∗ that maximizes the joint action value.

3.2 Value Decomposition inMARL

Value Decomposition (VD) is a useful technique in cooperative MARL to achieve effective Q-learning [Sunehag et al., 2018; Rashid et al., 2018; Wang et al., 2021; Li et al., 2021; Jiang et al., 2021; Qing et al., 2024]. It aims to learn a joint action-value function Qtot to estimate the expected return given current global state st and joint action ut. To realize VD, a mix network f(·; θυ) with parameters θυ is adopted as an approximator to estimate the joint action-value function Qtot . f(·; θυ) is introduced to merge all individual values into a joint one Qtot = f(q; θυ), where q = [Qn ]Nn=1 ∈ RN and Qn with shared parameters θa is the action-value network of each agent an. Usually, f(·; θυ) is enforced to satisfy the Individual-Global-Max [Son et al., 2019] principle. Therefore, the optimal joint action can be easily derived by independently choosing a local optimal action from each local Q-function Qn , which enables Centralized Training and Decentralized Execution (CTDE). The learnable parameter θ = {θa, θυ} can be updated by minimizing the TemporalDifference (TD) loss as:

where E[·] denotes the expectation function, D is the replay buffer of the transitions, ytot = r +γ ˆQtot is the one-step target and ˆQtot is the target network [Mnih et al., 2015]. Additionally, owing to the fact that the partial observability often limits the agent in the acquisition of information, the agent policy usually uses past observations from history [Sunehag et al., 2018].

3.3 Policy Gradient inMARL

Policy gradient (PG) [Yu et al., 2022; Lowe et al., 2017; Foerster et al., 2018] has been proposed as a competent alternative to directly optimize the policy. In the domain of cooperative MARL, the PG mechanism complies with the CTDE constraint through the learning of an individual actor πn (un |τ n ), and a centralized critic V (s) : S → R, for all agents. To leverage global information during centralized training, the value functions V n typically incorporate the global state s as input to ensure accurate estimation of the expected value. By adopting this decentralized execution approach, the consequent implicit joint policy achieves a fully independent factorization and agent policies are assumed to be independent of each other [Fu et al., 2022]: π(u | τ) = ∏Nn=1 πn (un | τ n ) . PG directly maximizes the expected discounted return Rt = ∑∞i=0 γ i rt+i as the objective. Thus, the loss function is defined as Lactor = -Eπ[Rt]. To optimize the actor π, we can perform policy gradient [Sutton and Barto, 2018] as:

where pπ is the state distribution. In particular, Rt is often replaced by rt + γV (st+1) - V (st) which is calculated by the critic function to reduce the high variance.

4 Method

To introduce global cooperative information for agent training, we propose the Centralized Advising and Decentralized Pruning (CADP) framework, as shown in Figure 2. In general, CADP performs CTDE to enable each agent to learn its individual policy network. At the heart of our design is introducing the explicit cooperative information exchanging of agents for sufficient centralized training to enhance the agent policy network. To cope with the decentralized execution paradigm, we design a model self-pruning mechanism, which prompts the centralized model to evolve into a decentralized model smoothly. The overall framework is finally summarized. Besides, for the sake of clarity, we employs the VD method as an illustrative example to introduce the proposed CADP framework, while CADP is also readily applicable to the PG method.

4.1 Advice Exchanging

The widely adopted CTDE framework only introduces the global state for agents in the mix/critic module, leading to that an agent policy network only perceives its local observation instead of the global states. In contrast, we design a novel centralized training scheme to augment the agent policy network from the local information of an individual agent to the global cooperative information from all agents, inferring better action decisions.

Formally, we employ an agent's confidence c for all agents to highlight its personalized confidence weights of other agents when receiving interchangeable cooperative advice from them, where the higher confidence corresponds to the more useful information of agents. The whole process can be reduced to a self-attention mechanism [Vaswani et al., 2017], where we set messages key k and value v, respectively, while confidence c is considered as the dot product of the key k of other agents and query q of itself. q, k and v are all linear projections of the local observation o. The formula is written as:

where kj means the message key k of agent j and qi means the query q of agent i, dx is the scaling coefficient and ci,j is the confidence from agent i to agent j. Finally, the aggregating information zi of the agent i is obtained by taking the weighted sum of the value according to the confidence

where vj means the value v of agent j. Through this step, each agent refers to the cooperative information of others. Then, we combine the aggregating information z in the previous step with the agent's own local information h, and finally output the action value Q:

where GRU(·, ·) stands for Gated Recurrent Unit. Notably, we have incorporated residual connections into the input of the GRU network. This short-circuit mechanism allows us to simultaneously leverage representations with and without advice exchanging, for enhancing training stability and performance.

Since our work focuses on the agent policy module, we can adopt different mix modules such as VDN [Sunehag et al., 2018], QMIX [Rashid et al., 2018] and QPLEX [Wang et al., 2021] to generate Qtot . Besides, if the final output of our agent module is a policy distribution πi instead of Qi (performing normalization after Equation 5), we can also employ MAPPO [Yu et al., 2022].

4.2 Model Self-Pruning

In pursuit of facilitating decentralized execution, the current model needs to evolve into the decentralized model depending only on itself, rather than global information. In this step, we design a simple yet effective model self-pruning method to achieve this. We claim that if the following conditions are satisfied, the agent model is a decentralized model:

where ei means the i-th standard basis vector (an one-hot vector). In this way, we can directly apply the agent model to decentralized execution as there is no advice from other agents used. Furthermore, we just apply the value v to produce the output z without key k and other components. For the convenience of expression, we refer to the model that only requires their own values v without the self-attention mechanism as the decentralized model (CADP (D)). On the other hand, the model using self-attention mechanism is referred to as the centralized model (CADP (C)). A decentralized model actually presents that the agent's confidence ci,1:N of all agents is equal to the one-hot vector ei in the execution stage. It is required to smoothly swap from the centralized training with exchanging confidence to the decentralized execution with the one-hot confidence ci,1:N. Therefore, we design an auxiliary loss function named pruning loss Lp to help the decentralized agent gradually alleviate the dependence of other agents, which is given as:

where θa means the parameters in the agent module. In the pruning loss, smaller Lp means agents rely less on the others.

4.3 Overall Framework

In our framework, we adopt advice exchanging step at agent policy module to produce more thoughtful and teamoriented action decisions. After the model achieves satisfactory performance, we start to use the pruning loss Lp, prompting the centralized model to evolve into a decentralized one smoothly. Our mix module implementation uses QMIX [Rashid et al., 2018] as a basic backbone for its robust performance and its simplicity of architecture, but it is readily applicable to the other mix/critic method since our framework focuses on the agent policy module.

To sum up, training our CADP framework contains two main loss functions. The first one is naturally the original TD loss LTD mentioned on Equation 1, which enables each agent to learn its individual agent policy by optimizing the joint-action value of the mix module. Unlike IGM-DA [Hong et al., 2022] and CTDS [Zhao et al., 2022], to avoid performance degradation and reduce computing costs, our pruning loss Lp is not introduced at the very beginning. We add pruning loss when the centralized model reaches a high level. Therefore, the total loss of our framework is formulated as follows. (Here we take the method of value decomposition as an example. For the PG method, we just need to add the pruning loss to the loss of actor LAC mentioned on Equation 2):

where θmix stands for the parameters of mix network in our method and t is the timestep of the training. σ(·) is a threshold function which is defined as follows:

where T is a hyperparameter and α is the coefficient for trading off loss term. From a practical application perspective, it suffices to incorporate the pruning loss after the centralized model achieves satisfactory performance in online learning. Nevertheless, in Section 5.4, we still conduct ablation experiments for T to further investigate its impact. In addition, we provide pseudocode in Appendix D.

5 Experiments

To demonstrate the effectiveness of the proposed CADP framework, we conduct experiments on the StarCraft II micromanagement challenge and Google Research Football benchmark. We aim to answer the following questions: (1) Can CADP outperforms the methods under traditional CTDE framework? (Section 5.1) (2) Can CADP outperforms the methods under teacher-student CTDE framework? (Section 5.2) (3) Can CADP perform better than the message (communication) pruning methods under the CTDE constraints? (Section 5.3) (4) Can CADP, as a universal framework, be applied to different methods and achieve improvement? (Section 5.4) (5) In comparison to traditional CTDE methods, does CADP require excessive time overhead? (Appendix C). Besides, visualization is given in Appendix E.

Based on these questions, we opt to compare three major categories of baseline methods: traditional CTDE-based methods, teacher-student CTDE framework methods and the message pruning methods under the CTDE constraints. Each category will be compared with CADP. Our CADP framework uses QMIX [Rashid et al., 2018] as a basic backbone for its robustness and its simplicity of architecture, but it is readily applicable to the other mix/critic backbone. We show it at Section 5.4. The detailed hyperparameters are given in Appendix B , where the common hyperparameters across different methods are consistent for comparability. We also provide additional expereiments in Appendix B.

5.1 Comparison with Traditional CTDE

We select several traditional CTDE methods: VDN [Sunehag et al., 2018], QMIX [Rashid et al., 2018], QTRAN [Son et al., 2019], QPLEX [Wang et al., 2021], CWQMIX [Rashid et al., 2020], OWQMIX [Rashid et al., 2020] and one DTDE-based methods (regarded as a special case of CTDE): IQL [Tan, 1993] for comparison. The experimental results on different scenarios are shown in Figure 3. It can be seen that our proposed method successfully improves the final performance in the challenging tasks. Especially in the most difficult homogeneous scenario (3s5z vs 3s6z) due to the different unit types, the large number of entities and the great disparity in strength between the two teams, our method can outperform baselines by a large margin.

5.2 Comparison with Teacher-Student CTDE

We also compare with the teacher-student CTDE framework methods: CTDS [Zhao et al., 2022] and IGM-DA [Hong et al., 2022]. The experimental results on different scenarios are shown in Figure 3. Similarly to the previous section, our method has better final performance in the challenging tasks, not only compared with the methods under teacherstudent CTDE framework. Especially in the super hard scenarios (3s5z vs 3s6z) and (corridor) due to the different unit types or the large number of entities and the great disparity in strength between the two teams, our method outperforms baselines by a large margin. In addition, after adding the pruning loss, the performance of our decentralized model rises dramatically from almost zero to near the performance of our centralized model. By contrast, in methods teacherstudent CTDE framwork, the performances of student models and teacher models climb together slowly and even sometimes the teacher models fail for training. Furthermore, our CADP method is the only method which can ultimately reach almost consistent performance for decentralized model and centralized model in all scenarios.

Besides, we test teacher-student CTDE framework methods on the Google Research Football (GRF) [Kurach et al., 2020] benchmark. Unlike StarCraft II, individual observations in the Google Research Football (GRF) are not partial observation, which contains as much information as global state. In this case, the teacher-student CTDE framework will no longer have a significant advantage over methods under traditional CTDE framework theoretically, since the teacher model does not have more input information than the student model and has no ability to instruct the student model. The experimental results on different scenarios are shown in Figure 4. It can be seen that only our method significantly outperforms QMIX [Rashid et al., 2018]. In addition, in the method of the teacher-student framework, the student model always maintains almost the same performance as the teacher model, which verifies that the teacher model does not have the ability to guide the student model in this case and also shows that using global cooperative information is more powerful than simply providing more observation information. In GRF benchmark, we set T = 3M in 3 vs 1 with keeper scenario and T = 6M in counterattack easy scenario respectively. We can see an obvious improvement of our method after adding the pruning loss Lp at timestep T in both.

5.3 Comparison withMessage (Communication) Pruning Methods under CTDE Constraint

Furthermore, we test some message pruning methods, GACML [Mao et al., 2020a], NDQ [Wang et al., 2020c], MAIC [Yuan et al., 2022] and MACPF [Wang et al., 2023] under CTDE constraint. GACML [Mao et al., 2020a] and MAIC [Yuan et al., 2022] are attention-based methods, while the other is not. During training, we allow them to communicate and apply message pruning, but during execution, we cut off all communication. To ensure a fair comparison, we also used QMIX [Rashid et al., 2018] as the backbone for these methods. The experimental results in the table 1 show that these methods are almost not better than basic QMIX [Rashid et al., 2018] which means when communication is completely cut off, the performance of these methods is not good.

These experimental results indicate that although message pruning can reduce communication, it is not suitable for CTDE scenarios where there is no communication at all. These message pruning methods, except MACPF [Wang et al., 2023] focus on compressing and refining communication information or selecting who to communicate with, which results in smaller bandwidth. As for MACPF [Wang et al., 2023], the poor performance can mostly be attributed to the insufficient information exchanging, which is solely the onehot vector representing the actions of the previous agents. In contrast, our CADP framework is designed to enhance the CTDE framework, which does not allow any communication at all during the decentralized execution phase. Our CADP framework first trains a well-performing communication model, and then gradually and dynamically distills its knowledge into a non-communication model.

5.4 Ablation Study

Different backbones. To further verify the generability of our CADP framework, we test value-based methods: VDN [Sunehag et al., 2018], QMIX [Rashid et al., 2018], QPLEX [Wang et al., 2021] and policy-based method: MAPPO [Yu et al., 2022] under the CADP framework. Experimental results in Table 2 indicate that all these methods have shown an improvement when using the CADP framework. Especially in the most difficult homogeneous scenario (3s5z vs 3s6z), methods with our CADP framework outperform baselines by a large margin. This observation indicates the versatility and effectiveness of our proposed CADP framework in catering to various CTDE-based methods, particularly in scenarios with high levels of difficulty.

Different hyperparameter. We examine the effect of the coefficient α in 3s5z vs 3s6z scenarios in Figure 5. We observe that there is not much difference in performance except the coefficient is too small (α = 0.001) to affect the training. Even when we set α = 100.0, The final performance is also not far behind the best (α = 0.5). This means our method is very robust for the coefficient α. In addition, we consider five T values: 0, 1M, 2M , 3M, and 3.5M. Figure 6 demonstrates that using smaller T for training, the performance of decentralized starts to improve earlier but taked longer time from the beginning of improvement to convergence which may waste some computing resources while using bigger T for training, the performance of decentralized model improves more quickly but the convergence time may delay. In general, setting T = 3M is a balanced choice for both sides. The experiment also shows that different values have little impact on the final performance, which may be because LTD is always dominant in training before the convergence of the centralized model.

6 Conclusion

In this paper, we argue the traditional CTDE framework is not centralized enough, since it falls short of fully utilizing global information for training, leading to an inefficient exploration of the joint-policy, and resulting in performance degration. Thus, we propose a novel Centralized Advising and Decentralized Pruning framework, termed as CADP, to enhance basic CTDE with global cooperative information. It is noting that our focus is not to design a new communication method. Our main contribution is adopting agent communication to enhance the basic CTDE framework for fully centralized training and still guarantees the independent policy for decentralized execution. As CADP provides a light yet efficient framework, we believe there will be more discussion and exploration under CADP framework.

Acknowledgments

This work was supported in part by the CCF-Baidu Open Fund under Grant No. CCF-BAIDU OF202410, in part by the Hangzhou Joint Funds of the Zhejiang Provincial Natural Science Foundation of China under Grant No. LHZSD24F020001, in part by the Zhejiang Province HighLevel Talents Special Support Program "Leading Talent of Technological Innovation of Ten-Thousands Talents Program" under Grant No. 2022R52046, in part by the Zhejiang Provincial Natural Science Foundation of China under Grant No. LMS25F020012, and in part by the advanced computing resources provided by the Supercomputing Center of Hangzhou City University.