PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learning

1 Introduction

Many real-world tasks can be modeled as decision problems for multi-agent systems, such as multi-robot navigation [Han et al., 2020], multi-robot collision avoidance [Long et al., 2018], multi-UAV path planning [Qie et al., 2019], information retrieval [Chen et al., 2024] and games [Mao et al., 2021; Guss et al., 2021]. In most of these scenarios, the prevalent constraints include partial observability, and agents are constrained to decentralized decision-making processes.

∗Corresponding author

To address these challenges, Multi-Agent Reinforcement Learning (MARL) has emerged as a focal point of research. Within MARL, Centralized Training with Decentralized Execution (CTDE) stands out as a prominent paradigm. CTDE leverages global information during training and shifts to utilizing only local information during execution, facilitating decentralized decision-making. This paradigm encompasses two primary algorithmic categories: value-decomposition based and actor-critic based approaches. Concerning the utilization of global information, the former category [Sunehag et al., 2017; Rashid et al., 2018; Wang et al., 2020; Chai et al., 2021] employs global information for enhancing the joint Qfunction. In contrast, the latter category [Lowe et al., 2017; Foerster et al., 2018; Iqbal and Sha, 2019; Yu et al., 2021; Zhang et al., 2023a] incorporates global information as input to a centralized critic. Notably, these approaches refrain from utilizing global information directly during execution, a factor that could potentially constrain collaboration performance among agents, especially in complex scenarios, as demonstrated in our experiments.

In contrast to the conventional CTDE approach, a distinct line of research explores the direct utilization of global information during execution. COPA [Liu et al., 2021] introduces a Coach-Player framework, devising an adaptive communication method wherein the coach determines when to dispatch a global instruction vector to the players. This vector, combined with local information, is used to compute individual Q-functions. Despite COPA's use of a multi-head attention mechanism for comprehensive global information processing during execution, this information remains identical for all agents. In a different vein, the CSRL framework [Chen et al., 2022] introduces a Commander-Soldiers MARL framework, incorporating the concept of specific information for each agent. Both COPA and CSRL notably enhance multiagent collaboration performance by directly applying global information during execution.

Nevertheless, practical challenges emerge in numerous applications due to local observability constraints, posing difficulties in utilizing global information directly during execution. To reconcile the need for global information while ensuring decentralized execution, we propose a novel paradigm named Personalized Training with Distilled Execution (PTDE), which comprises two training stages. In the first stage, we introduce the concept of agent-personalized global information by employing a Global Information Personalization (GIP) module. This module transforms raw global information into personalized global information tailored to each agent. This personalized global information is then utilized to compute individual Q-functions or individual policies, enhancing the performance of each agent. In the second stage, we implement knowledge distillation for the agent-personalized global information. Within this distillation framework, a proficiently trained GIP module acts as the teacher network, while a dedicated student network is employed for the distillation process. Crucially, the input for the student network is exclusively composed of the agent's local information, presenting a departure from the teacher network, which integrates both global and agent's local information 1 .

During execution, the teacher network is replaced by the student network, enabling decentralized execution while retaining the benefits of personalized global information. This innovative approach ensures a seamless transition from personalized training to distilled execution within the proposed PTDE paradigm. Summary of our contributions:

• In contrast to the prevalent trend in CTDE-based methods, which emphasizes leveraging global information during centralized training, our approach shifts the focus to exploring the utilization of global information during decentralized execution.
• We identify that consistently positive performance among agents is challenging when applying the same global information for decision-making. However, our innovation lies in transforming global information into agent-personalized global information, resulting in agents consistently making improved decisions.
• We introduce a novel paradigm named PTDE, which not only benefit from agent-personalized global information but also executes in a decentralized manner through knowledge distillation. Importantly, our experiments demonstrate minimal performance degradation after distilling agent-personalized global information into agent's local information.
• Experimental results underscore the universality and efficacy of the PTDE paradigm across diverse multi-agent environments and algorithms.

2 Background

2.1 Dec-POMDP

In this work, we model a fully cooperative multi-agent task as the Dec-POMDP [Oliehoek and Amato, 2016], which is formally defined as a tuple G = 〈S,U, P, r, Z,O, n, γ〉. s ∈ S is the global state of the environment. Each agent i ∈ A ≡

1 Our approach diverges from the conventional knowledge distillation employed in model compression [Gou et al., 2021], where both the teacher and student networks operate on the same input.

{1, ..., n} chooses an action ui ∈ U which forms the joint action u ∈ U ≡ U n . The state transition function is modeled as P (s′ |s, u) : S ×U× S → [0, 1]. The reward function which is modeled as r(s, u) : S × U is shared by all agents and the discount factor is γ ∈ [0, 1). It follows partially observable settings, where agents do not have access to the global state. Instead, it samples observations z ∈ Z according to observation function O(s, i) : S×U → Z. Each agent has an actionobservation history trajectory τ i ∈ T ≡ (Z × U)∗ , on which it conditions a stochastic policy πi (ui |τ i ) : T × U → [0, 1]. In our algorithm, the joint policy π is based on action-value function Qπ (st, ut) = Est+1:∞,ut+1:∞[∑∞k=0 γ k rt+k|st, ut]. The final goal is to get the optimal action-value function Q∗ .

2.2 Typical MARL Algorithms

Value decomposition [Sunehag et al., 2017; Rashid et al., 2018; Wang et al., 2020] and Actor-Critic [Yu et al., 2021; Zhang et al., 2022; Zhang et al., 2023b; Hu et al., 2024] are two typical branches of multi-agent reinforcement learning. Among these, VDN [Sunehag et al., 2017] is the representative algorithm to formulate value-decomposition paradigm. QMIX [Rashid et al., 2018] learns a monotonic factorisation ensuring that a global argmax operation on the joint actionvalue function Qtot yield the same results as a series of individual argmax operations on each individual action-value function Qi. QPLEX [Wang et al., 2020] takes a duplex dueling network architecture to factorize the joint value function. MAPPO [Yu et al., 2021] is an actor-critic based algorithm. To specialize for multi-agent settings, MAPPO uses the structure of PPO algorithm but the critic can take extra global information to follow the CTDE framework.

2.3 Knowledge Distillation

Knowledge distillation [Hinton et al., 2015] is proposed to compress big models. It distills the knowledge generated from a larger network into a smaller network. Policy distillation [Rusu et al., 2015] presents a novel knowledge distillation method which can be used in reinforcement learning to extract the policy of agent and train a new network with an expert level performance and better efficiency. CTDS [Zhao et al., 2022] proposes a novel Centralized Teacher with Decentralized Student framework which consists of a teacher model and a student model to alleviate the inefficiency caused by the limitation of local observability.

3 Method

In this section, we initially present an approach that provides the same global information to all agents during execution for decision-making. Despite its simplicity, this naive use of global information does not consistently enhance multiagent collaboration performance. Subsequently, we propose the Global Information Personalization (GIP) module to tailor global information for each agent, resulting in the agentpersonalized global information. Based on this, we derive a centralized execution method that makes better use of global information for improved performance. Recognizing the challenge of directly obtaining global information during execution, we finally introduce the knowledge distillation approach to achieve decentralized execution without too much performance degradation.

3.1 Naive Use of Global Information

We present a method that directly employs unified global information for all agents during execution. Using QMIX as an example, the entire framework is depicted in Figure 1 (a). (1) The RNN module encodes the trajectory of the agent's local information Oti = (oti , ut-1 i , idx) to htoi .

1. The Global Information Unification (GIU) module is designed to generate the unified global information sti to be used in execution, where the green multilayer perceptron (MLP) encodes the raw global information st into hst and the blue module transforms hst into a multivariate gaussian distribution N ∼ (μsti , Σsti ).

(3) Similar to QMIX, the individual action-value Qti (τ t i , uti ) is computed by an MLP operating on the concatenation of htoi and sti , while the joint action-value Qttot (τ t , ut ) is computed by nonlinearly combining all individual action-values through the mixing network.

Since the parameters of GIU module are unified and invariant to each agent during execution, the algorithm is called QMIX GIU (Global Information Unification). It is one of the baselines and ablations in our experiments.

3.2 Global Information Personalization

In many multi-agent cooperative tasks, an agent's decisionmaking is significantly improved by concentrating on a subset of the global information, given that the raw global information tends to be redundant [Mao et al., 2020]. Extracting and utilizing this relevant portion of global information is crucial for optimal decision-making. Inspired by this observation, we introduce the Global Information Personalization (GIP) module, crafted to autonomously tailor the global state (i.e., extracting the beneficial part) for each individual agent.

As shown in Figure 2, the GIP module comprises three components: Agent-Hyper Network, Agent-Personalization Network, and Distribution Generator. The Agent-Hyper Network takes the agent's local information as input and produces a set of weights W and biases B. The structures of the Agent-Personalization Network and Distribution Generator are identical to those in Figure 1 (b). The output of the GIP module, denoted as zt i , is defined by Equation (1).

Compared to the GIU module (i.e., Figure 1 (b)), a distinguishing feature of GIP module is that the parameters of Agent-Personalization Network are dynamically generated by Agent-Hyper Network. Since the local information is different for each agent, the parameters of AgentPersonalization Network are guaranteed to be personalized to each agent, which is the key to global information personalization. Therefore, we call zt i the agent-personalized global information in this paper.

The GIP module is versatile across existing CTDE algorithms. In Figure 3 (a), the integration of the GIP module in value-decomposition methods is illustrated. Here, the individual Q-function is computed by concatenating htoi and zt i . Figure 3 (b) showcases the application of the GIP module in actor-critic methods, where the concatenation [htoi , zt i ] is utilized by the individual actor for action sampling.

3.3 Knowledge Distillation

The methods shown in Figure 3 involve the utilization of the global state st when computing individual Q-functions or individual policies (as zt i depends on st). However, obtaining global information directly is challenging due to partial observability in real-world multi-agent systems. To leverage global information during execution while adhering to the need for decentralized execution, we introduce a knowledge distillation method. This approach distills agent-personalized global information using only the agent's local information, i.e., transforming the dependence of zt i on st into the practical reliance on (oti , ut-1 i , idx).

The knowledge distillation framework is illustrated in Figure 4. In this setup, the GIP module serves as the teacher network, while the student network is represented by an MLP. As indicated in Equation (4), the calculation of zt i involves st and Oti = (oti , ut-1 i , idx) through the teacher network. On the other hand, the student network's input consists solely of the agent's local information Ot i , and its output is denoted as zt i ′ in Equation (5). In the context of knowledge distillation, zt i is referred to as teacher knowledge, and zt i ′ is student knowledge. Equation (6) illustrates the use of Mean Squared Error (MSE) Loss to minimize the disparity between student and teacher knowledge, ensuring effective training of the student network.

Knowledge distillation is a common technique employed for model compression, typically involving identical input data for both teacher and student networks. In our knowledge distillation, a notable distinction arises as the student network's input lacks global information st compared to the teacher network. This divergence from traditional model compression is pivotal, serving as the key factor in transitioning from centralized execution to decentralized execution.

Upon completion of the knowledge distillation training, the student network can seamlessly replace the teacher network (i.e., the GIP module) during the execution process. In other words, zt i ′ is utilized in lieu of zt i in Figure 3, enabling decentralized execution.

3.4 The Overall PTDE Paradigm

Figure 5 shows the overall framework of PTDE based on the QMIX algorithm, encompassing two-stage training and decentralized execution.

The First Training Stage. We provide personalized global information for each agent to compute a better individual Qfunction or individual policy. Specifically, the RNN module records the trajectory of agent's local information and outputs an encoding vector htoi . The teacher network gives agent-personalized global information zt i . Then, the individual action-value Qti (τ t i , uti ) can be computed by concatenation [htoi , zt i ]. Finally, the joint action-value Qttot (τ t , ut ) can be obtained by the mixing network. The whole network is trained end-to-end by minimizing the loss shown in Equation (7) and (8):

where b is the batch size of sampled experiences from replay buffer; ϕ is the parameters of the teacher network and θ is parameters of the other networks in the first training stage; θ- and ϕ- are parameters of target networks. We also show the pseudo-code in Algorithm 1.

Algorithm 1: The first training stage of PTDE

Initialize: The parameters θ and ϕ of network, θ- and ϕof target network, replay buffer D. Initialize: Observation o = (o1, · · · , oN) and state s. while not over do Collect a tuple (o, s, a, r, o′ , s′ ) by generating 8 parallel episodes, and store it in D; Sample a random minibatch b from D; The RNN calculate htoi , the teacher network calculate zt i ; Calculate ytot and loss L for all sampled data from b based on Equation (8) and (7); Update the parameters of networks θ, ϕ; Update the parameters of target networks θ-, ϕ- every N episodes; Output: Get a well-trained teacher network and an algorithm that works well to execute centrally.

The Second Training Stage. After the first training stage, the teacher network can provide agent-personalized global knowledge zt i for agents' decision-making. In the second stage, we use offline knowledge distillation where the student network distills teacher knowledge zt i to obtain student knowledge zt i ′ . Subsequently, the student network can replace the teacher network during the execution process. The pseudo-code can be seen in Algorithm 2.

Decentralized Execution. As shown in Figure 5 (b), agents utilize solely local information (oti , ut-1 i , idx) to compute individual action-values Qti (τ t i , uti ) and sample actions using uti = argmaxuQti (τ t i , uti ). We name this method as QMIX KD.

(a) Two-stage training

(b) Decentralized Execution

4 Experiments

Our experiments mainly focus on six research questions:

• (RQ.1) Does the utilization of unified global information during execution lead to an enhancement in the performance of multi-agent collaboration?
• (RQ.2) Is agent-personalized global information more effective in improving performance compared to unified global information?
• (RQ.3) After knowledge distillation, can the algorithm maintain a substantial portion of its performance when transitioning from centralized to decentralized execution?
• (RQ.4) Does the PTDE-based algorithm exhibit universality across diverse environments?
• (RQ.5) Does the PTDE paradigm demonstrate universality across various algorithms?
• (RQ.6) What is the rationale behind the PTDE paradigm's approach of partitioning the training process into two stages?

We investigate the research questions using popular MARL testbeds, namely StarCraft II [Samvelyan et al., 2019] and Google Research Football [Kurach et al., 2020]. Additionally, we validate RQ.4 through experiments in the Learning to Rank (LTR) scenario [Liu and others, 2009]. To our best knowledge, this is the first time that the MARL algorithm has been applied to LTR tasks.

For baselines, we categorize them into two classes: centralized execution algorithms and decentralized execution algorithms, outlined in Table 1. To showcase the impact of agentpersonalized global information, we contrast our approach with two centralized execution algorithms, CSRL and COPA, and perform an ablation experiment using QMIX GIU. Furthermore, to assess algorithm performance after knowledge distillation, we utilize QMIX and QPLEX as decentralized execution baselines. All experiments are conducted using the PyMARL2 framework [Hu et al., 2021] with 8 parallel runners and 3 random seeds. Details regarding hyperparameters are available in Table 7 in the Appendix.

4.1 StarCraft II

To address RQ.1, RQ.2, and RQ.3, we select hard scenarios such as 5m vs 6m and 3s vs 5z, as well as super hard scenarios like 3s5z vs 3s6z and 6h vs 8z. Additionally, to further highlight the advantages of agent-personalized global information in multi-agent collaboration, we conduct experiments in two new scenarios, namely 3s vs 8z (featuring 8 zealots in the enemy team) and 3s5z vs 3s7z (with 3 stalkers and 7 zealots in the enemy team), where existing decentralized execution algorithms exhibit poor performance.

As shown in Table 2, QMIX GIU has better performance than QMIX in 3s vs 5z, 5m vs 6m and 6h vs 8z, but performs worse in 3s vs 8z and 3s5z vs 3s6z. This addresses RQ.1, indicating that unified global information can have a positive impact on decision-making in certain scenarios but may not consistently improve agent decisions. In contrast, QMIX GIP consistently outperforms QMIX GIU in all testing scenarios, supporting RQ.2 by highlighting the consistent benefits of agent-personalized global information. Furthermore, QMIX GIP attains the highest winning rates among all centralized execution algorithms, further affirming the advantages of agent-personalized global information (RQ.2).

The Performance Retention Ratio (PRR) in Table 2 signifies the ratio of the winning rates of QMIX KD to those of QMIX GIP, reflecting the performance before and after knowledge distillation. The PRRs range between 85% and 90% in four out of six simulation maps, indicating that the PTDE paradigm can maintain performance reasonably well after knowledge distillation (RQ.3). Notably, even in challenging scenarios like 3s5z vs 3s7z, where all baselines have low winning rates, QMIX KD achieves a winning rate of 63.1%, showcasing the substantial advantages of PTDE in such extreme conditions. Overall, the PTDE paradigm's ability to train a viable strategy with the assistance of agentpersonalized global information and subsequently achieve decentralized execution through knowledge distillation is highlighted. The training curves (Figure 9 in Appendix C) and strategy visualizations (Appendix A) also illustrate the performance improvement of QMIX KD over QMIX.

4.2 Google Research Football

To further validate RQ.1, RQ.2, and RQ.3, we select five widely recognized academy scenarios: 3 vs 1 with keeper, 3 vs 2 with keeper, counterattack easy, counterattack hard and run pass and shoot with keeper. The agents are trained for 10 million steps using 8 threads in all scenarios.

Table 3 displays winning rates on GRF. QMIX GIP achieves the best performance across all scenarios, reinforcing the notion that agent-personalized global information is more beneficial for multi-agent collaboration than unified global information (RQ.2). The high PRRs further demonstrate that the performance does not degrade significantly after knowledge distillation (RQ.3). Specifically, in 3 vs 1 with keeper and run pass and shoot with keeper, PRRs range from 95% to 100%, while in counterattack easy and counterattack hard, PRRs range from 80% to 90%. These conclusions align with the training curves available in Figure 10 in Appendix C.

4.3 Scenario Universality of PTDE Paradigm

To further validate the effectiveness of the PTDE paradigm, we extend its application to the Learning to Rank (LTR) [Liu and others, 2009] task. Ranking plays a crucial role in information retrieval, where the goal is to arrange a list of candidate documents in descending order of relevance to a given query. Achieving an optimal search ranking list enhances the effectiveness of information retrieval.

In the multi-agent cooperation setting for the Learning to Rank (LTR) task, each document is treated as an agent. The fundamental components of Multi-Agent Reinforcement Learning (MARL) are defined as follows:

• Observation: Features of the query and document i.
• State: Features of the query and all documents.
• Reward: Given that NDCG@k [J¨arvelin and Kek¨al¨ainen, 2002] is a standard evaluation metric for ranking, we define the reward as NDCG@k.
• Action Space: Discrete scores, such as integers from 0 to 9, where the document is ultimately sorted based on the scores assigned to each document (agent).

We conducted training and testing on 10,000 queries (7:3 partition) from the MSLR-WEB30K [Qin and Liu, 2013] dataset, a large-scale dataset for Learning to Rank research. We modified this dataset to comprise 10 documents per query, resulting in a total of 10,000 queries and 100,000 documents. The experimental results are depicted in Figure 6. Our approaches, QMIX GIP and QMIX KD, achieve higher NDCG scores compared to QMIX and QMIX GIU, addressing RQ.2 and RQ.3. Interestingly, QMIX GIU performs worse than QMIX, providing additional insights into RQ.1.

Crucially, the experiments in LTR, along with those on the StarCraft II and GRF benchmarks, collectively demonstrate that the PTDE paradigm exhibits good universality across diverse scenarios, providing a response to RQ.4.

4.4 AlgorithmUniversality of PTDE Paradigm

In this section, we integrate the PTDE paradigm with VDN and MAPPO, and test them on the 3s vs 5z and 3s5z vs 3s6z scenarios. We adopt hyperparameters as specified in [Hu et al., 2021] and [Yu et al., 2021], respectively. As shown in Table 4 and 5, VDN GIU's performance on 3s5z vs 3s6z is inferior to that of VDN, and MAPPO GIU's performance on 3s vs 5z is worse than MAPPO. This highlights that unified global information does not always contribute to the efficacy of multi-agent collaboration (RQ.1). Furthermore, VDN GIP outperforms VDN GIU, and MAPPO GIP outperforms MAPPO GIU in both scenarios, demonstrating the effectiveness of the PTDE paradigm for both valuedecomposition-based algorithm VDN and actor-critic-based algorithmMAPPO. In essence, the PTDE paradigm exhibits good universality across different algorithm types (RQ.5). The training curves for Table 4 and Table 5 can be found in Figure 11 in Appendix C.

4.5 Empirical Analysis of the Two-Stage Training

Why is it necessary to divide the training process into two stages? In Table 6, we compare the PTDE and CTDS paradigms based on two metrics: winning rate and PRR. CTDS synchronizes the distillation of global policies with centralized training, while our PTDE approach first conducts centralized training and then performs agentpersonalized global knowledge distillation. Across the 5m vs 6m, 6h vs 8z, and 3s5z vs 3s7z maps, the PRR metric consistently favors the PTDE paradigm over the CTDS paradigm. This underscores that the two-stage training approach of PTDE maintains superior performance during decentralized execution (addressing RQ.3 and RQ.6).

Figure 7 displays the loss function (6) and test winning rates during the knowledge distillation process on the 3s vs 5z scenario. The loss plot displays training and testing loss curves spanning 0 to 40k steps. The winning rate plot showcases a blue line representing test winning rates for decentralized execution after knowledge distillation, and a red dashed line signifying the test winning rate for centralized execution before knowledge distillation. In this experiment, the batch size is set to 1000, and each step corresponds to the training of one batch. As can be observed, a notable reduction in loss occurs before 30k steps, accompanied by a swift increase in the test winning rate. This suggests that distilling global policies using local information requires a specific training duration. Conversely, simultaneous knowledge distillation during centralized training might fail to preserve optimal performance in decentralized execution due to inadequate training and uncertainties in the distribution of training samples. This analysis addresses and responds to RQ.6.

5 Conclusions

We introduced a two-stage training paradigm, named Personalized Training with Distilled Execution (PTDE), designed for multi-agent reinforcement learning. In the first training stage, the Global Information Personalization (GIP) module tailors global information for each agent. Subsequently, during the second training stage, the student network distills agent-personalized global information using solely the local information of each agent. In the execution stage, the student network takes over from the teacher network, enabling decentralized execution.

Our empirical evaluations on the StarCraft II benchmark, Google Research Football benchmark, and Learning to Rank task collectively offer conclusive answers to the posed research questions (RQ.1 to RQ.6). These results provide robust evidence supporting the efficacy and broad applicability of the PTDE paradigm.

Acknowledgments

This research was supported by the Natural Science Foundation of China (61902209, 62377044, U2001212), and Beijing Outstanding Young Scientist Program (NO.BJJWZYJH012019100020098) and Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China.