Dynamic Belief for Decentralized Multi-Agent Cooperative Learning

1 Introduction

Many complex sequential decision-making problems [Chen et al., 2017; Foerster et al., 2018] require all agents to achieve a unified goal or the largest team utility in a decentralized way, where all agents coordinate their behaviors conditioned only on their own observation history. Due to the high-dimensional dynamic state space and unknown environment model, deep multi-agent reinforcement learning (MARL) shows great potential and has attracted a lot of interest in these years.

To learn decentralized policies, a typical baseline [Tan, 1993; de Witt et al., 2020] is to develop an independent learner for each agent and regard the collective (or global) rewards as the individual rewards directly. This paradigm may be faced with the non-stationarity [Bowling and Veloso, 2002] problem: the dynamics of its environment effectively changes as allies (or teammates) change their behaviors through learning and agents may receive spurious reward signals that originate from their allies' behavior. To make the learning stable, most of the existing methods typically adopt the paradigm of centralized training and decentralized execution (CTDE) [Rashid et al., 2018], which assume the learning occurs in a laboratory or a simulator where the extra global state and communication are available. Despite the great progress it has made, the CTDE paradigm still may be limited in some realistic multi-agent systems due to the difficulty of developing enough real simulators and the inaccessibility of the global states or agents' communication for training. Take the autonomous vehicles [Cao et al., 2012] for example. Even though RL models have been deployed on vehicles in the factory before delivery, they still require further learning in the real road environment where every vehicle is independent without centralized scheduling. Therefore, it calls for a more practical decentralized approach with partially observed setting both in training and execution, where every agent learns to cooperate only by its own observation without global information as well as the observation and policies of others.

∗Corresponding author.

To develop a stable decentralized training method, existing independent learning handles the non-stationarity by elaborating training practices like clipping PPO policy ratio [Sun et al., 2022] and repeating Q-learning updates[Abdallah and Kaisers, 2016], or improving experience replays[Palmer et al., 2018][Lyu and Amato, 2018] . Different from them, our method handle the problem in an other-agent-modeling manner, in which latent behaviors of the other agents is considered for learning. Related to our work, some methods of agent modeling [Hernandez-Leal et al., 2019] [Hong et al., 2017] learn a static policy belief model to predict and respond to the actions of other agents. However, they assume fixed policies of other agents and fail to apply on the decentralized multiagent learning in which the policies of other agents continue updating by learning itself, as illustrated in Fig. 1. In this situation, a desirable capability of agents is to dynamically change their policy beliefs over the development of others' policies. A naive way for the purpose is to fine-tune the policy belief model only utilizing the most recent action histories, considering that the histories in more previous episodes are out-of-date to the current policies of others. However, such approaches may suffer a lot from the lack of data and insufficient training and thus prevent the policy belief from modeling others effectively. In the subsequent part of this paper, belief refers to policy belief for clarity.

In this paper, we propose to learn a dynamic belief for dynamically modeling other agents over their changing policies in learning and accordingly condition its policy. To deal with the developing policies of other agents, we meta-train the dynamic belief to learn to model other agents by only a few action histories in the most recent episodes, as illustrated in Fig. 1(b). Our learned belief model of each agent takes as input: (i) the current observation, and (ii) a set of recent observations and observed actions of other agents. Then it outputs an embedding of belief to be used by prediction of the future actions of others. Specifically, we introduce a historical context to dynamically process the recent histories relevant to the current state by essential use of soft attention. And then, we present a latent variational inference to produce the belief embedding by modeling policies of others as a distribution with VAE [Pu et al., 2016]. Our method is implemented based on the Proximal Policy Optimization (PPO) [Schulman et al., 2017] structure, which has shown impressive success in many reinforcement learning tasks. Trained by predicting actions of other agents, the belief embedding of an agent then conditions both the policy decision and the value computation in the training as well as the execution stages. Note that our method is not to seek the optimal policy of an agent based on the policies of other agents, but to learn effective policies of all agents simultaneously under decentralized learning framework. We take belief as an additional input when optimizing the policy function, which avoids the instability caused by recursive predictions.

In summary, our contribution is as follows: (i) We proposed a novel dynamic belief learning to alleviate the nonstationarity of decentralized multi-agent learning by modeling the changing policies of teammate agents and conditioning the policy of its own. (ii) It introduces a historical context and a latent variational inference which learn to dynamically model other agents by only a few action histories in the most recent episodes. (iii) Experiment results on two cooperative environments demonstrate that our approach improves the performance over other independent learning methods in decentralized settings and even achieves a comparable performance to CTDE.

2 RelatedWork

Multi-agent cooperative learning. Although deep RL has been well explored using centralized controllers, those methods designed for single agents cannot be applied for multiagent RL tasks since the joint action space grows exponentially with agent numbers. Hence, centralized training and decentralized execution (CTDE) was proposed as a popular paradigm for MARL [Kraemer and Banerjee, 2016]. Typically, value-based approaches learn the global stateaction values as the aggregation of individual state-action values. For instance, VDN [Sunehag et al., 2017] and QMIX [Rashid et al., 2018] produce the global Q-values by an arithmetic summation or a non-linear monotonic factorization. Other works further extend the class of value functions, i.e., QTRAN [Son et al., 2019]. Another typical approach, Actor-Critic MARL, learns independent actors by a centralized critic using policy gradients, such as MAPPO[Chao et al., 2021][Sun et al., 2022], COMA[Foerster et al., 2018], MADDPG [Lowe et al., 2017] and so on. Despite the significant progress that has been made, CTDE may still be limited in some realistic multi-agent tasks due to the requirement of centralized training. Therefore, the practical application calls for a decentralized approach with partially observed settings both in training and execution. Under this limit, independent learning (IL) like IPPO[de Witt et al., 2020], IQL [Tan, 1993] and IAC [Foerster et al., 2018] decompose an nagent MARL problem into n decentralised single-agent problems and directly learn decentralized policies for each agent. However, they often suffer from the non-stationarity of the environment induced by agents simultaneously learning and exploring, making them unable to learn optimal policies in some environments. In this paper, we explore a different potential to alleviate non-stationary by learning and exploiting a dynamic belief of others.

Other agent modeling. Modeling and adapting to unknown other agents is crucial for multi-agent environments, where agents interact simultaneously. With access to opponents' trajectories, learning other agent adaptively is widely used in multi-agent competitive tasks [He et al., 2016] or human-ai coordination [Carroll et al., 2019]. ToM [Rabinowitz et al., 2018], ToMoP [Yang et al., 2019] consider a purely observational setting and predicted other agents' desires, beliefs, and intentions from the observed state and action. InAC[Ma et al., 2021a] learns interactions in a centralized manner. Furthermore, several methods [Papoudakis and Albrecht, 2020] [Zintgraf et al., 2021] model others' polices as a latent distribution by variational autoencoder (VAE) [Pu et al., 2016]. Differently, our work assumes partially observed training settings with only access to the observation of its own. PR2[Wen et al., 2019], GR2[Wen et al., 2021] learn recursive reasoning of others to seek optimal responding while our method is to learn effective policies of all agents simultaneously. NRBS[Moreno et al., 2021] learns recursive belief for policy training but ignores relative historical observations. OP[Ma et al., 2021b] collects and matches the behavioral graph of opponents directly as policy inputs, whereas our work mines historical samples to learn belief by action prediction. Related to our work, previous methods [Hernandez-Leal et al., 2019] [Hong et al., 2017] model other agents by learning a static belief to predict their future actions assuming fixed policies of other agents, similar to imitation learning [Duan et al., 2017]. However, such methods fail to apply to the decentralized learning setting in this work, where agents change their policies by learning simultaneously. MeLIBA [Zintgraf et al., 2021] assumes a prior distribution over others' policies but still can't tackle our settings without prior distributions. For this situation, our method dynamically changes the belief over the development of others' policies by the recent historical observations and effectively helps the learning of policy.

Transformer in RL. Transformer [Vaswani et al., 2017] has been employed in many recent RL methods [Janner et al., 2021] [Parisotto et al., 2020] [Banino et al., 2021] [Meng et al., 2021]. A typical line of methods used a transformer to replace GRU/LSTM for modeling temporal information over the sequence of states [Chen et al., 2021] [Parisotto et al., 2020] [Banino et al., 2021], while another line of methods employed the architecture among multiple agents to exploit the inherent relation across agents for multi-agent scenarios [Meng et al., 2021] [Hu et al., 2020] [Inala et al., 2020]. Different from them employing transformers to choose actions, our work introduces it to learn the belief about other agents by mining historical samples. Significantly, the attention in our work is applied to different objects from them, neither between different states in the same episode for temporal relations nor between different agents in the same state for inherent interactive impacts. It is between the current state and those in historical episodes to explore the relative samples as references for dynamic belief inference.

3 Background

Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). [Oliehoek and Amato, 2016] Consider a fully cooperative multi-agent task that can be described as a stochastic game G, defined by a tuple G =< S,U, P, r,Z,O, n, γ >. s ∈ S describes the true state of the environment. At each time step, each agent a ∈ A ≡ {1, ..., n} chooses an action ua ∈ U, forming a joint action u ∈ U ≡ U n , which induces a transition in the environment according to the state transition function P(s′ |s, u) : S×U×S → [0, 1]. All agents share the same reward function r(s, u) : S ×U → R and γ ∈ [0, 1) is a discount factor. This paper considers a decentralized multi-agent learning setting in both training and execution. Every agent is trained individually without sharing policy model or replay buffer with each other. We consider a partially observable scenario in which each agent draws individual observations z ∈ Z according to observation function O(s, a) : S × A → Z. Each agent has an action-observation history τ a ∈ T ≡ (Z × U)∗ , on which it conditions a stochastic policy πa (ua |τ a ) : T ×U → [0, 1]. MARL agents aim to maximize the discounted return Rt =

∑∞l=1 γ l rt+l. Throughout this paper, we omit the superscript a to represent the current agent for clarity. Quantities in bold represent joint quantities over agents, i.e., u, and bold quantities with a tilde such as ˜u denote joint quantities over agents other than the current one.

Action-observable Setting. Our work assumes that actions of the observed other agents are observable, which is practical and reasonable in realistic applications. Taking autonomous vehicles for example, the moving and steering actions of other vehicles can be easily observed.

Proximal PolicyOptimization (PPO). The PPOalgorithm [Schulman et al., 2017] is one of the most popular singleagent RL methods due to its advanced performance, stable training, and easy implementation. PPO learns a policy function πθ with parameter θ and a value function Vϕ(s) with parameter ϕ, where Vϕ(s) is used for variance reduction and only utilized during training. To optimize πθ, PPO imposes the policy constraint through a clipped surrogate objective function:

where rθ = πθ(u|s) πθold (u|s) , θold is the previous parameter in rollout. and ϵ is the clipping bound. Noting that PPO is an actorcritic algorithm, the policy π is the actor and the advantage function A is the critic. The advantage function represents how good a state-action pair is compared with the average value of current state, i.e., A(s, u) = Q(s, u) - V (s), where Q(s, u) is the action-value function estimated by samples and V (s) is the approximation of the state-value function. PPO uses the generalized advantage estimation (GAE) to compute the advantage, which uses the linear combination of n-step bootstrapping to obtain low bias and low variance.

Independent PPO (IPPO). InMARL settings Independent PPO (IPPO) [Chao et al., 2021] decomposes a multi-agent problem into a collection of simultaneous single-agent problems that share the same environment. Each agent learns policy πθ and state-value function Vϕ of itself independently without sharing information such as observations or replay buffers. This approach considers other agents as a part of the environment and thus suffers from the non-stationarity of training,

due to the changing policies of other learning agents, ˜u ̸= ˜u′ . Therefore, it could not guarantee convergence even in the limit of infinite exploration.

4 Methods

Since each agent receives the joint reward responding to the actions of all agents, it is hard to learn π(ut|ot) from r(st, ut, ˜ut) directly because the uncertainty of other agents' actions ˜ut. To handle this challenge, we introduce belief δt as an embedding vector corresponding to ˜ut, and aim to learn π(ut|ot, δt) alternatively. Note the state transition relies on all agents' actions in Dec-POMDP, and the observation of each agent belongs to the global state. Hence, the agent's observation history τt is related with other agents' actions, especially which are close with current agent. Hence, δt could be generated by τt, and further be used to predict ˜ut. To effectively learn δt where other agents learn simultaneously, we first introduce the dynamic belief learning for dynamically modeling other agents over their changing policies, and then present how to extend the single agent RL method PPO to Belief PPO for decentralized multi-agent cooperative learning.

4.1 Dynamic Belief Learning

In principle, with the progress of learning, the previous interaction history can no longer reflect the policy information of other agents, except the most recent ones because of the continuity of learning. To this end, we propose dynamic belief learning, which meta-trains a model to learn to predict belief only by the action histories of other agents in the most recent episodes. In each step, our dynamic belief network receives as input (i) the current observation, and (ii) historical observations and the observed actions of other agents in the recent episodes. Then the dynamic belief network produces an embedding of belief to be used by predicting the future actions of the teammates and the policy of itself. As illustrated in Fig. 2, the proposed architecture is composed of two modules: the historical context and the latent variational inference, where the former is used to learn context embedding from historical steps in the recent episodes and the latter presents the belief δa t by sampling from a learnt latent distribution.

Historical Context

Historical context utilizes soft attention over the most recent historical steps to dynamically learn belief. Firstly, an observation embedding network fo and an action embedding network fa are introduced. They receive an observation or a one-hot encoded action as input and produce an embedding of the observation or the action. In each step t, the current observation ot and the historical observations {ˆ oˆ t , ˆ t = 1, ..., Th} are encoded by embedding network fo to produce a current observation embedding fo(ot) and a sequence of historical observation embedding {fo(ˆ oˆ t ), ˆ t = 1, ..., Th} respectively, where ˆ t is the time step in the history and Th is the length of the used history. For dynamic belief inference, the input history can span hundreds to thousands of time steps, and training with such long sequences for every current step can be demanding in both time and memory usage. Considering that most historical steps are far irrelevant to the current states, we use an adaptive dropout operation to discard a subset of irrelevant time steps during both training and execution. The adaptive dropout depends on the cosine similarity S between the sequence of historical observation embedding and the current observation embedding,

where Kot represents the subset of top-Nh historical steps selected specifically for the current observation ot. We denote p as the proportion of time steps that are preserved, and Nh = pTh.

After downsampling the historical steps, the historical context processes embedding of both the current state and historical states, including observations as well as the observed others' actions, and outputs a context embedding. Hence, it is forced to capture only the relevant information, which will be used to produce the belief embedding and predict the actions of others. To this end, we make use of soft attention between the current state and historical states. The module starts by computing a query vector q(ot) as a function of the current state embedding fo(ot), which is then used to attend over the different states in the most recent histories. To capture sufficient information of the historical states, we compute the state embedding of historical steps as the aggregation of the observation embedding and the action embedding by element-wise adding fs(ˆ sˆ t ) = fo(ˆ oˆ t ) + fa(ˆ˜ uˆ t ). The action embedding is produced by a linear layer on the one-hot encoded action ˆ˜ uˆ t of other agents. Then the key vector k(ˆ sˆ t ) and the value vector v(ˆ sˆ t ) are computed for every historical state as different functions of the historical state embedding. The context embedding is aggregated by adding the current state embedding and the weighted summary of the value vectors over different historical states,

where wi is the attention weight computed by the scaled inner product between the query vector and the key vectors: wi =< q(ot), k(ˆ sKot [i]) >.

Latent Variational Inference

Since the policies of other agents are not fixed, we assume other agents' policies as a Gauss distribution N(µt,σt) where µt and σt represent the mean and variance respectively. Therefore, an encoder function is introduced following the soft attention over observation module to predict µt and σt respectively, and then the belief is sampled from N(µt,σt). To make the sampling derivable to µt and σt, the re-parameter trick [Kingma and Welling, 2013] is used:

where δI is sampled from the standard normal distribution.

Based on δt, a decoder function composed by a linear layer is used to predict teammates' actions. Consider the observed teammate's actions ˜ut as the sampled datapoints and the learning objective could be formulated as maximizing the likelihood function max log(p(˜ut)). However, log(p(˜ut)) could not be optimized directly. Similar to ELBO [Kingma andWelling, 2013], its lower bound is:

where DKL is the KL-divergence and p(δt|fctx) = N(µt,σt). Q(δt) is the prior probability distribution and defined as the standard normal distribution in our method. Hence, maximizing log(p(˜ut)) could be translated to maximizing the lower bound. Specifically, maximizing p(˜ut|δt) is equal to minimizing the mean square error:

where ˜upt is the predicted actions of other agents and vt ∈ {0, 1}n-1 is the visibility for other agents obtained from the environment for ignoring the prediction error of the invisible teammates.

4.2 Belief PPO

Here, we introduce how the proposed dynamic belief learning can be incorporated into the PPO algorithm and achieve effective decentralized multi-agent cooperative learning. We follow the algorithmic structure of PPO by learning a policy network πθ and a state-value function network Vϕ(s) for each agent. Since making decisions only by the individual observation of an agent will lead to non-stationarity of training, we make the policy and the value function condition on both the observation and the learned belief embedding about others. The policy and the value function share the same belief embedding within each agent for information reuse and robust training. Specifically, in each step t, the policy network starts by processing the current observation ot with a base network and then combines the information of the current state and the inferred belief through a belief fusion module. In our experiments, the belief fusion is implemented by concatenating the belief embedding from the dynamic belief network and the state embedding from the base network and then applying to it a linear transformation. After that, the final policy with the size of action space is produced by another linear layer. The state-value function network is of the same structure as the policy but has a different output size of 1. During training, the policy network, the value function network, and the dynamic belief network are optimized simultaneously by the PPO loss and belief loss according to Eq. 1 and Eq. 6,

where α and β are the weight factors of MSE and KLdivergence, respectively.

5 Experiments

We evaluate our approach on a fully cooperative environment SMAC [Samvelyan et al., 2019], a standardized decentralized StarCraft II micromanagement environment. In each scenario, algorithm-controlled independent ally units fight against enemy units controlled by the built-in game AI. And an episode of game is declared a victory only if all enemy units are eliminated. In the decentralized setting, each agent can only achieve partial observation within its sight range, including location, health, and actions taken by visible agents. Any global information used by CTDE methods is unused.

Details. During training, 8 paralleled episodes are rolled out independently to generate data. The most recent 800 steps before last optimization are used for reference histories in dynamic belief network. And p is set to 0.05 for the adaptive dropout. The loss weights α = 0.5 and β = 0.005.

5.1 Comparison with Other Methods

Comparison with Decentralized Methods. We compare the overall results of our approach with three independent learning approaches: IPPO, IQL, and IAC in the same decentralized settings. Note that our implementation of IPPO is equivalent to SOTA independent method PG-ind[Fu et al., 2022] in ICML2022. The results in three maps are illustrated in Fig. 3(a-c). Independent learning methods like IPPO, IQL, and IAC fail to learn policies that consistently defeat the enemy, especially in the map of 3s vs 5z. Moreover, the performance over training steps is volatile due to the non-stationarity of the environment as other agents learn to change their policies during training. However, our approach achieves far higher performance than those methods in the same decentralized settings without sharing information. The superior results demonstrate that our approach alleviates the disadvantages of non-stationarity by the dynamic belief, which effectively anticipates the actions of others.

Comparison with CTDE Methods. The state-of-the-art CTDE methods that train agents in a centralized way are also compared, including MAPPO, QMix, COMA, RODE[Wang et al., 2020b] and QPLEX[Wang et al., 2020a]. Among them, MAPPO with feature-pruned global states and the fine-tuned QMix [Hu et al., 2021] with elaborate optimizations perform the best in SMAC. As shown in Table 1, these methods perform well due to extra global state information and centralized training to address non-stationarity. Nevertheless, our approach achieves comparable and even superior performance with them in most scenarios, which indicates its capability to effectively utilize the limited partial information. However, our method performs not well in 6h vs. 8z for two reasons: 1) Global states are significant for such complex scenario while our method dose not use them. Specifically, the controlled units in 6h vs 8z are much weaker (with lower armor and slower speed) than enemies compared with other maps, which is difficult and demands more cooperation among agents. With the help of the extra global state and agents' communication (i.e., the share of model and replay buffer), MAPPO learns better cooperation and outperforms our method. Under the decentralized setting, MAPPO is equivalent to IPPO and is outperformed by us with a large margin as shown in Fig. 3, which demonstrates that the superiority of MAPPO to our method is heavily attributed to the global states and agent's communication. 2) Since the scenario is hard to explore[Hu et al., 2021], our decentralized setting is limited by lower data efficiency, about only 1/6 of CTDE, because only the trajectories of an agent's own are used for training. As shown in Fig. 4, our method achieves comparable performance eventually as the training steps increase. And for other easy-to-explore scenarios such as corridor, our method achieves faster improvement, demonstrating its superiority in learning ability without the limitation of exploration. Moreover, the approach is more practical without centralized restriction.

Comparison with Agent Modeling. We compare our method with related agent modeling methods: DRPIQN [Hong et al., 2017] and AMF-A3C[Hernandez-Leal et al., 2019], which also learn networks to predict and respond to teammates' actions. Specifically, we re-implemented AMFA3C based on a better baseline structure of PPO. As shown in Fig. 3(d), both the methods perform not well because they learn a static belief designed for fixed teammates and thus fail to tackle the decentralized learning where other agents also change their policies during training. Our dynamic belief outperforms them by large margins, thanks to quickly adapting to the development of teammate policies.

5.2 Ablation Studies

Component effects. To validate the effect of each component, ablations are studied in detail on the environment of the SMAC corridor map, as shown in Fig. 5(a). With IPPO as baseline, we first evaluate the static belief by directly learning an embedding that predicts the actions of others and then conditions the policy. As shown in Fig. 5(a), the performance drops distinctly compared to the baseline due to it being incapable of predicting the changing policies of other agents. Then we evaluate the latent variational inference (LVI) by applying it to learn a distribution of latent variables for the belief, denoted as Belief with LVI. The experiment improves the performance of static belief and surpasses the baseline, demonstrating the effect of LVI to process the uncertainty of others' policies and alleviate adverse effects of mispredictions. Furthermore, we validate the effectiveness of the historical context. With the context module, the overall dynamic belief achieves a rapidly growing win rate and outperforms the Belief with LVI by large margins. The improvements can be explained by the soft attention, which helps dynamically predict the updating policies of others. It effectively learns to infer their policies only from a few historical steps in the most recent episodes. We also validate the embedding observation for dropout by replacing it using raw observation as shown in Fig. 5(b), which fails to learn effective policy. Our method calculates the distance of historical observations on their embedding because the raw observation space is not discriminative, which directly contains the values of health, position and ignores importance of every element. And the embedding network automatically learns to project the raw observation into a latent space, where semantically similar observations have smaller distance. Since the embedding network is only a 2-layer MLP, the additional computational cost is low and affordable.

Generalization. To validate the generalization of the method, we evaluate it on a non-fully cooperative environment traffic junction [Sukhbaatar et al., 2016], where each agent needs to maximize independent rewards by coordinating with others. In the traffic junction, cars enter a junction from all entry points with a probability of parr. Every car can take two actions at each time-step, gas and brake respectively. For traffic junction, an episode is considered a failure if a collision happens. The success rate refers to the percentage of episodes without failure. We adopt the medium level of difficulty as described in [Sukhbaatar et al., 2016]. As shown in Table 2, consistently improved results can also be observed in the traffic junction environment. It demonstrates the general effectiveness of the proposed methods for different environments in decentralized settings.

Parameter analysis. Ablations of history length Th in historical context and preservation ratio p in the adaptive dropout are evaluated in Fig. 5(c-d). The results show that a larger Th is likely to converge faster and perform better, while smaller Th can still achieve comparable results eventually with less computation. We use Th = 800 for both performance and complexity. Ablations of p (for Top-N) show that the method cannot learn effective policies when p is too small which abandons useful histories, while it may converge slowly when p is too large. The method performs best when p = 0.05.

Importance of belief over training. To validate the importance of belief over training steps, we evaluate the sum of weights in the belief fusion network, which takes as inputs both the belief embeddings and current observation. The sum ratio of weights for belief to all weights (for both belief and current observation) is shown in Fig. 5(e). The ratio increases after an initial decrease, indicating that the belief has less influence on policy at the beginning and becomes more important to it over training. Specifically, when the teammates take random actions in the initial learning, the dropout approach is hard to preserve valuable historical observations, since even similar observations may have different teammates' actions. Hence, their actions cannot be accurately predicted for producing belief embeddings. Hence, the policy may pay more attention to the current observation than the belief to learn general actions without relation to others. As training goes on, since the teammates' actions are relatively deterministic and predictable, the belief embeddings will learn valuable information and play a more important role in the policy.

6 Conclusion

In this paper, we introduced dynamic belief learning for decentralized multi-agent cooperation, which relieves the nonstationarity by dynamically modeling the changing policies of other agents. Extensive experimental results demonstrate that our method outperforms independent learning approaches and even achieves comparable performance with CTDE methods.

Acknowledgements

This work is partially supported by grants from the Key-Area Research and Development Program of Guangdong Province under contact No. 2020B0101380001 and grants from the National Natural Science Foundation of China under contract No. 62027804, No. 61825101 and No. 62088102. The computing resources of Pengcheng Cloudbrain are used in this research.