Diversifying Training Pool Predictability for Zero-shot Coordination: A Theory of Mind Approach

1 Introduction

Building social agents that can coordinate with novel agents is challenging [Stone et al., 2010; Dafoe et al., 2021; Mirsky et al., 2022; Treutlein et al., 2021; Bowling and McCracken, 2005; Bard et al., 2020] because artificial social agents need to adapt to diverse partners. To overcome this challenge, recent works generate a pool of diverse artificial agents and train the adaptive agent to cooperate with agents from this pool. This kind of generative framework to create high-quality training data [Jiang et al., 2023] is actively studied in deep learning [Zha et al., 2023b;

Whang et al., 2023; Zha et al., 2023a; Jakubik et al., 2022; Mazumder et al., 2022] and benefits different sub-fields such as computer vision [Paul et al., 2021] and natural language processing [Yu et al., 2022b; Brown et al., 2020]. In the context of building adaptive agents, manipulating the process of generating the training policies provides the advantage of instilling different prior knowledge and inducing flexibility in cooperating with the partner's behaviour in the training pool. To create a pool of diverse agents, existing works have studied various schemes to generate pools that contain agents with varying trajectories [Lupu et al., 2021; Lou et al., 2023; Zhao et al., 2023], skills [Szot et al., 2023], or desires [Yu et al., 2022a].

In this paper, we explore a new diversity scheme based on the predictability of others in the team in training adaptive agents in a zero-shot coordination setting. If an agent is only trained with collaborators whose behaviours have high predictability, it will develop specific habits to cooperate with this population and not readjust to novel partners, i.e. will struggle when its new partners are less predictable. Negative effects also can happen when the adaptive agent is only trained with agents that are completely random in behaviour, i.e. low in behavioural predictability, and bring no benefit to the team since this can potentially lead the adaptive agent to learn to achieve the task individually without cooperating with others. Hence, we suggest maintaining agents with different levels of behavioural predictability (LoBP) in the population with which we train the adaptive agent. An agent or a policy is unpredictable if its behaviour cannot be predicted from a third-person point of view-the theory of mind observer-, and vice versa 1 . We continually train a theory of mind observer to predict the behaviour, including one-stepahead primitive actions and intentions, of self-play agents to assess the behavioural predictability of agents in a particular environment. Although recent research has considered machine theory of mind as a crucial component for artificial agents in cooperative multi-agent settings [Rabinowitz et al., 2018; Papoudakis et al., 2021], using this model to maintain the diversity of agents has not been studied.

Our training framework is presented in Fig. 1. In the first stage, a training pool of diverse agents is developed via selfplay reinforcement learning (RL), and we pick the agents for a training pool for the next stage based on LoBP-based selection criteria. In the second stage, the adaptive agent is trained to cooperate with the agents in this training pool. We focus on the first stage, where the training pool is generated to have diverse LoBP. Our proposed approach includes (1) a theory of mind observer, which assigns a behavioural predictability score for agents in the pool; and (2) an episodic memory, which supports the construction of the training pool with diverse LoBP and favours agents that have meaningful behaviours that appeared in the later stage of training. We conducted experiments on the Overcooked environment [Carroll et al., 2019] and demonstrated that the LoBP-based mechanism of selecting a training pool helps the adaptive agent learn to cooperate better with unseen partners compared to other diversity-based baselines.

To summarise, our contributions are:

• A generative training framework with a theory of mind observer and episodic memory that employs the behavioural predictability to create a pool of diverse agents and is helpful to train the adaptive agent;
• Empirically showing our method produced the adaptive agents that outperform state-of-the-art methods on different cooperative tasks when paired with novel partners;
• An in-depth analysis of the pool of agents developed during the selection process.

2 RelatedWorks

Adaptability of reinforcement learning agents in social setting and zero-shot coordination. Creating a pool of agents and training an adaptive agent to best respond to this pool is emerging research to achieve zero-shot coordination. Approaches following these lines attempt to address the problem of coordinating with humans without collecting and using human data [Strouse et al., 2021]. This is opposed to traditional approaches [Carroll et al., 2019] where the adaptive agents should be trained with human proxies before cooperating with humans, which is often expensive. The automated process of generating a pool of artificial agents often benefits from approaches that encourage diversity, either by diversifying the population during training or by collecting agents. Towards enhancing the ability to generalise for artificial intelligence, this focuses on addressing the question of determining the data on which deep neural networks should undergo training to achieve effective generalisation [Yarats et al., 2022; Jiang et al., 2023], i.e. highlighting the role of training data to the adaptability of agents. In other words, if the model is exposed to diverse datasets, it can generalise and behave better in testing with situations that are unseen during training.

1 We note that in case the actual entropy of agents' behaviour can be computed, the maximum behavioural predictability which is extensively studied in human mobility research [Song et al., 2010; Lu et al., 2013] can also be used as an alternative criterion within our framework.

Maintaining diversity in the self-play pool. The first work proposed in this line is [Strouse et al., 2021], in which the pool of agents is constructed by selecting agents at three different stages of learning. This method used self-play reinforcement learning [Tesauro, 1994; Silver et al., 2018] to train independent self-play agents. After the training process, the self-play agents selected for the pool are diverse in their performance on the task. They have (a) low performance, i.e. at the beginning of training; (b) high performance, i.e. at the final stage of training; and (c) medium performance, i.e. have the rewards are the average of high and low performance. Although this is a simple approach, the adaptive agent trained on this pool surprisingly behaves well when coordinating with novel partners. Yu et al. [2022a] employed the human bias into the pool of agents via hand-crafting a reward system used in training, i.e. the desire or motivation of agents. For each self-play agent, the reward is computed by weighting pre-defined events. The weights are then randomised so that agents in the pool have different motivations during training. We note that the event-based reward system in this work is built based on human judgement of events in scenarios of Overcooked. Hence, this approach is specifically designed for the Overcooked benchmark and requires human effort to construct the diversity scores.

Maintaining diversity in the population-based training. Compared to self-play training, population-based training [Jaderberg et al., 2017] allows more control over the population of collaborators to develop the pools. Direct measurement on the output of policies such as the Jensen-Shannon divergence is employed to create a strategy that obtains the population with agents that have diverse trajectories (TrajDiv) in [Lupu et al., 2021]. In MEP [Zhao et al., 2023], the entropy rewards were derived to encourage the agents to behave not only differently from others in the population but also differently from themselves. Utilising the intrinsic rewards for conditional policy to act differently according to different discrete inputs was common used in single reinforcement learning [Eysenbach et al., 2018]. Szot et al. [2023] adopt this technique to create a pool of agents that has a specific identity and agents are trained to have different behaviours (BDP).

In [Lou et al., 2023], the adaptive agent is trained frequently with versions of the policies in the pool during the training period. This method classified agents during the training process into different sets based on their performance to evenly select agents. All the listed methods do not explore the use of behavioural predictability as a criterion to select a diverse set of agents in the training pool.

Modelling other agents and machine theory of mind. Understanding others is a crucial ability for artificial agents to engage in social interactions [Albrecht and Stone, 2018]. This ability to attribute mental states, widely known as theory of mind in psychology research [Premack and Woodruff, 1978; Baron-Cohen et al., 1985], has recently been studied in AI [Rabinowitz et al., 2018]. Based on the assumption that novel partners seen during execution also are predicting others' behaviour, in [Knott et al., 2021], the adaptive agent is trained to cooperate with a theory of mind-based policy that is cloned from human data. However, the machine theory of mind has not been used to maintain the diversity of behavioural predictability in the pool of training agents as in our work.

3 Preliminaries

3.1 Notations

Let us consider the multi-player Markov Decision Process (MDP) [Boutilier, 1996] as a tuple M = ⟨N , S, A, P, R, γ⟩ where N = {1, . . . , N} is the set of N agents that operate in the scene, S is the state space, A = A1 ∪· · ·∪Ai · · ·∪AN is the joint action space and Ai is the action space of the agent i, P : S ×A×S ↦→ [0, 1] is the transition probability function, R is the reward function and γ is the discounted factor.

In this paper, we focus on tasks that involve two agents. For simplicity, we call them the adaptive agent, which is the agent that we can train to control during execution, and the collaborator, which is its partner that the adaptive agent needs to adapt to. It is worth noting that the adaptability of AI agents can be characterised as the ability to adapt to a variety of factors, such as environmental changes or objective changes. In our work, we consider the adaptive agent in social interactions, i.e. cooperating with different collaborators. At each time step t, each agent i will observe the state st ∈ S to take action ait ∈ Ai accordingly. The joint action of agents is denoted as at ∈ A, and the trajectory of an agent at each time step t is τt = (s0, a0, . . . st, at). Here, we study teaming in cooperative tasks; hence, the team will receive the team rewards rt ∈ R, with R : S × A ↦→ R.

3.2 Zero-shot Coordination

The aim of a zero-shot coordination (ZSC) algorithm is to find a policy πadapt : S ↦→ Ai for the adaptive agent to cooperate well with unseen agents. Formally, let us consider a set C of all collaborators with their policies denoted as πcolab . This set consists of two sub-sets Ctrain which contains agents seen by the adaptive agent during training and Chold-out which is the set of agents used during evaluation. When the adaptive agent is paired with the collaborator πcolab ∼ Chold-out sampled from the hold-out set, it acts to obtain the expected return of the team, hence, the objective of the adaptive agent πadapt is

where T is the length of an episode. The adaptive policy that the zero-shot coordination algorithm seeks when trained with only collaborators sampled from Ctrain is

3.3 Tackling Zero-shot Coordination via Generative Training Framework

To tackle the zero-shot coordination problem, recent works proposed using a generative strategy in training the adaptive agent [Strouse et al., 2021; Lupu et al., 2021; Szot et al., 2023]. This training scheme has two stages as follows. In the first stage, a set (pool) of collaborators Ctrain are generated. Since this set of collaborators is only used in training the policy of the adaptive agent πadapt , we denoted it as TrainingPool. In the second stage, the policy of the adaptive agent πadapt is trained to cooperate with agents selected from this TrainingPool to achieve cooperative tasks. If the training pool is diverse, the policy of the adaptive agent πadapt found in the second stage can cooperate with novel partners in evaluation.

We follow this two-stage training framework to train the adaptive agent. One distinct advantage of this framework is controlling the training data's diversity and quality. The generative procedure in the first stage allows us to manipulate or select the set of collaborators so that the adaptive agent can meet a diverse set of partners during training.

4 Proposed Approach

4.1 Overview

Our approach focuses on the first stage, i.e. generating the pool of training agents, of the two-stage training framework described in Section 3.3. Each agent in the pool is a policy which is parameterised by the weights, denoted by ψ. Given the set of Ne initial weights {ψi0 }i=1...Ne , we aim to generate a new set of weights called TrainingPool = {ψh}h=1...H, i.e. a pool of agents, with H is the size of the pool. This pool of H agents is then used to train an adaptive agent to cooperate with different types of novel partners.

We define the TrainStep(·) as a procedure that updates weights ψk of Ne agents, i.e. {ψik+1 }i=1...Ne ← TrainStep({ψik }i=1...Ne ). Iteratively executing this training procedure will create different versions of agents. We denote the number of iterations as K. We note that the TrainStep(·) procedure can follow any tradition of updating the weights and go through multiple update steps. Here, we choose to use the RL self-play procedure [Silver et al., 2018]. However, we do not restrict our approach to select agents induced by this type of policy training; therefore, one can explore using a population-based procedure under our framework.

We propose an algorithm that alternates between training and selecting the training weights to construct the TrainingPool. This is done by a theory of mind model and episodic memory. Briefly, at each iteration, we update Ne weights and store them in the memory via LoBP-based updating mechanism. So, our episodic memory keeps growing until we have H agents. After this, we replace agents to keep the memory size as H. We continually train a ToM model f T oM θ (·) to predict the behaviour of policies on behavioural data collected from the population. Then, the behavioural predictability of each agent is computed based on comparing the prediction of the ToM model with the ground truth behaviour. Finally, we construct the episodic memory that uses

Algorithm 1: Constructing the Training Pool (Fig. 2)

Input : {ψi0 }i=1...Ne is the set of Ne randomly initialised weights; ToM Agent f ToM θ (·); M is the episodic memory; K is the number of iterations. Output: TrainingPool = {ψh}h=1...H with H is the number of collaborators in the training pool. 1 {Di0 }i=1...Ne ←Roll-out({ψi0 }i=1...Ne ) ; 2 Dtrain 0 ,Deval 0 ← {Di0 }i=1...Ne ; 3 {σi0 }i=1...Ne ←ToMEval(Deval 0 ) ; 4 M ← UpdateMemory(M, {⟨ψi0 , σi0 ⟩}i=1...Ne ) ; 5 for k ← 0 to K -1 do 6 Run the training procedure {ψik+1 }i=1...Ne ←TrainStep({ψik }i=1...Ne ) ; 7 // LoBP Assessment (4.2): 8 {Dik+1 }i=1...Ne ←Roll-out({ψik+1 }i=1...Ne ) ; 9 Dtrain k+1 ,Deval k+1 ← {Dik+1 }i=1...Ne ; 10 θk+1 ← ToMUpdate(f ToM θ (·), θk;Dtrain k+1 ) ; 11 {σik+1 }i=1...Ne ←ToMEval(Deval k+1 ) ; 12 // Update Episodic Memory (4.3): 13 UpdateMemory(M, {⟨ψik+1 , σik+1 ⟩}i=1...Ne ) ; 14 end 15 Collect all weights inMto the TrainingPool ; 16 return TrainingPool

this information to collect the weights to the training pool. The proposed episodic memory stores pairs of the weight and its LoBP score.

The framework is shown in Fig. 1, and the overall algorithm is shown in Algo. 1. Details on the behavioural predictability and the ToM observer are presented in Section 4.2, and the episodic memory with its updating mechanism is introduced in Section 4.3.

4.2 Theory of Mind Observer and the Assessment of Behavioural Predictability

Behavioural Predictability

In this section, we will present how to compute the behavioural predictability of a policy, given information about the trajectories generated by this policy. This quantity is computed by a theory of mind observer, which is the model f T oM θ (·) parameterised by the weight θ. The ToM observer is trained to predict the actions and intentions of others. Given a weight ψik which represents the version k of the agent i in the pool, we collect the behavioural data Dik by rolling out the policy with respect to the weight ψik in the environment to get M trajectories, then query the prediction about the nextstep action and intention (˜at+1, ˜ gt+1) = f ToM θ (τt) from the theory of mind agent. The behavioural predictability of the policy with respect to the weight ψik is computed by

where

are the cross-entropy losses of the ToM model in predicting next-step primitive actions la and in predicting goals or intentions lg, respectively. Intuitively, if the trained observer does not predict well the behaviour generated by a policy (la or lg is high), then this policy can be considered as having a low behaviour predictability (σ is low), and vice versa.

According to our criterion, two arbitrary agents x and y with weights ψx and ψy , respectively, are considered as equal/similar in terms of LoBP if the gap between their predictability scores σx and σy is sufficiently small, i.e. |σx - σy | < δBP for a small δBP ∈ R+. We note that different approaches can be used to achieve diversity in behavioural predictability. One can build an agent model specifically for solving the task and use parameters to vary the predictability of agents in the pool. Due to simplicity, we propose to use episodic memory to promote LoBP diversity in Section 4.3.

Training the Theory of Mind Observer

In our approach, the ToM observer is continually trained along with the development of weights {ψik+1 }i=1...Ne . First, by rolling out policies with weights {ψik+1 }i=1...Ne in the environment, we collected the behavioural data {Dik+1 }i=1...Ne as shown in the 8th line of the Algo. 1. Second, this behavioural data obtained from the roll-out procedure {Dik+1 }i=1...Ne is divided into two sets Dtrain k+1 and Deval k+1 by a fixed ratio (Alg. 1, line 9th ). Finally, the ToM observer is trained on the training set Dtrain k+1 (ToMUpdate(·) in 10th of Alg. 1). It then computes the behavioural predictability σik on the evaluation set Deval k+1 (ToMEval(·) in line 11th of Alg. 1).

4.3 LoBP-based Episodic Memory

Design of LoBP-based Episodic Memory

We design the LoBP-based memory as an episodic memory M that contains H slots (equal the size of the TrainingPool) and is organised into B segments. Each slot stores a pair of weights and the behavioural predictability score ⟨ψ, σ⟩.We refer to this pair as the value of the memory slot, which can be updated, i.e. written or erased. Each segment S ∈ {S1, . . . , SB} contains several slots with a similar level of behavioural predictability as defined in Section 4.2. The number of segments B and the LoBP interval δBP are empirically chosen and fixed during the process.

Maintaining Diverse Level of Behaviour Predictability and Favouring Complex Behaviour

In this paper, we use a simple yet efficient mechanism to update the episodic memory and discard seen policies to guarantee a pool of agents with diversity in LoBP. The memory update mechanism, i.e. UpdateMemory(·) procedure, is described in Algo. 2. In detail, when a new policy with its behavioural predictability score, i.e. ⟨ψik , σik ⟩, comes to the update mechanism of the episodic memory, we will first erase the earliest memory value (making the slot empty) in the segment that has largest size (line 6th of Algo. 2), then write the weight and its predictability score ⟨ψik , σik ⟩ to the related segment. To guarantee there is no bias in the identity of the agent, we shuffle the agent set as in the 1st line of Algo. 2.

Algorithm 2: UpdateMemory

Input : {ψik }i=1...Ne is the set of Ne weights; M is the episodic memory. Output: M 1 for i in shuffle({1 . . .Ne}) do 2 ifMis not full then 3 M← ⟨ψik , σik ⟩ ; 4 else 5 Discard the earliest came value ⟨ψ, σ⟩f to the segment that has the maximum size Sm, i.e. ⟨ψ, σ⟩f ∈ Sm s.t. Sm = argmaxSj (|Sj|) where Sj ∈ {S1, . . . , SB} is the segment and |Sj| is the size of Sj; 6 M← ⟨ψik , σik ⟩ ; 7 end 8 end 9 return UpdatedM

Diversity in the level of behavioural predictability. To exploit all the capacity of the memory, when the memory has not reached its limitation of size, we will write the pairs of the policy's weight and its behavioural predictability score ⟨ψik , σik ⟩ into a slot in corresponding segments without any selection. When the episodic memory is full, to maintain the diversity in the LoBP and avoid overloading, we erase one value of the slot in the segment with the maximum size and write to the corresponding segment. We recall from the previous section that each segment in the episodic memory contains weights with similar levels of behavioural predictability.

Favouring complex and meaningful behaviour. Because behaviour displayed in the late stage of training is often more complex and meaningful than in the early stage, the update mechanism will keep later samples and discard earlier samples frommemory. In other words, this mechanism results in first-in-first-out (FIFO) property for each segment.

5 Experiments

5.1 Experiment Setting

We conducted experiments on different scenarios of Overcooked which is a benchmark to assess the zero-shot coordination ability of the adaptive agents. In this game, there are two agents that need to collaborate to achieve a cooking task. The Overcooked game was first introduced to study the coordination of AI in [Carroll et al., 2019]. Each agent can take one action in the set of 6 actions A = {up, down, left, right, interact}. During one episode, agents try to deliver as many dishes as possible before the end of the episode, reaching an end after T = 400 timesteps. To make one dish, agents have to collect ingredients and put them into the pot to cook. After 20 time steps of cooking on the stove, the dish is ready and can be delivered to a counter located at a fixed place in the room. In our experiments, all scenarios (Asymmetric Advantages, Coordination Ring) have sparse rewards, i.e. the team reward is only given to agents at the delivery time (successfully achieving the task).

5.2 Baselines

We consider state-of-the-art baselines in two branches of approaches that employ the generative two-stage training framework. They are different in classes of training methods used in stage 1: (1) the self-play (SP) training and (2) the population-based training (PBT).

• SP training method: fictitious co-play (FCP) [Strouse et al., 2021] that used RL SP strategy to train and a rewardbased rule to select collaborators into the training pool.
• Population-based training method: (1) Trajectory Diversity (TrajDiv) [Lupu et al., 2021]; (2) Maximum entropy population-based training (MEP) [Zhao et al., 2023]; and (3) Behaviour diversity play (BDP) [Szot et al., 2023]. All three methods utilise PBT to develop the training pool but use different criteria for maintaining the diversity of the pool.

We consider two versions of our approaches where the LoBP score is only computed by action prediction, i.e. there is no lg in Eq. 1, and where both action and intention prediction are taken into account. To evaluate our approaches, we followed the literature on experimental designs. We first compare adaptive agents in cooperating with reinforcement learning agents that are developing via self-play procedure. We used different seeds to generate the pool of agents to train the agents used to evaluate the adaptive agents in the test phase. We then evaluate the coordination skill of adaptive agents with the scripted agents and the human proxies.

5.3 Cooperating with RL Agents

To evaluate the ability to cooperate with novel agents, we trained a new set of RL agents by self-play training. This set is trained with different seeds from the training process of SP agents in the FCP pool and our pool of agents. The new RL self-play agents used to evaluate the adaptive agents are randomly selected during SP training. Hence, although they shared the same training algorithm (RL self-play process), the sets of agents used in train (Ctrain) and test (Chold-out) are different. Our approach outperforms other baselines on this task. During SP training, the RL agents exhibit behaviours with diverse predictability while learning skills. Therefore, training by our approach helps social agents to adapt well to these behaviours. The performances when pairing adaptive agents with hold-out RL agents are shown in Table 1.

5.4 Cooperating with Scripted Agents

In this section, we constructed scripted agents that have high behaviour preferences. We considered interactions between the adaptive agent and three types of behaviour: (1) placing onion in pot and delivery when the soup is cooked; (2) delivery when the soup is cooked; and (3) putting onion everywhere. While the first two types of agents have meaningful behaviour, the third procedural agent's behaviour obstructs the team from achieving high performance. This type of useless predictable behaviour can appear in the RL selfplay training, however, they often in the early phase and will be discarded from the episodic memory due to the usefull predictable behaviour when the SP agents begin to learn nearoptimal behaviour. More visualisation of this phenomenon is shown in Fig. 4. The results are shown in Table 2.

5.5 Cooperating with Human Proxy Agents

General Human Proxy

We use the behaviour cloning method [Bain and Sammut, 1995] to train the policy πproxy that imitates the behaviour of humans in scenarios of Overcooked. These policies are referred to the human proxies. The human data used in our works are human interactions in experiments established in [Carroll et al., 2019]. The results are shown in Table 3 (performance of our method is higher and statistically significant different to performance of baselines with p < 0.05).

Specific Human Proxies

We also evaluate the adaptive agents on different human proxies. To construct different human proxies, we first metatrained a general proxy ¯πproxy on trajectories of 12 individuals. We then fine-tuned this ¯πproxy on trajectories of others 6 individuals to obtain the specific proxies for each of them {πproxy 0 , πproxy 1 , πproxy 2 , πproxy 3 , πproxy 4 , πproxy 5 }. This type of training is adopted from [Duan et al., 2017]. These policies are assumed to reflect the behaviour of individuals. The results are shown in Figure 3. Our approaches produce adaptive agents that gain higher team rewards on average when cooperating with these policies.

FCP

Coord. Ring

TrajDiv MEP

BDP

Our Our

(intent)

Table 4: Comparison of portions in the pool of agents. Our method selects a small number of random agents than FCP.

5.6 Analysis of the Pool of Agents

Agents selected by Level of Behavioural Predictability

In this section, we analyse in detail the pool of agents that are generated by using the LoBP criteria. Fig. 4(a) shows that all SP agents were chosen to the training pool. Our training pool contains agents that are diverse in the level of behavioural predictability (Fig. 4(b)). To analyse the difference between the FCP pool and the LoPB-based pool, we divided the training process of the SP agents into three stages. The early stage is the period before the SP agent obtains half of the maximum training rewards. Following this stage are the middle and high stages where the SP agents obtain higher rewards. Table 4 shows that our pool mainly contains the weights of agents in the middle and high stages of training, i.e. the number of high-performance and meaningful in our pool is higher than the FCP pool. This helps the adaptive agents focus more on learning to cooperate with the agents that deliver complex behaviour instead of random behaviour often appearing in the early training phase. To further discussion, training good ToM observer can improve the efficiency of our approach, i.e. the ability of the ToM observer also affects to the settings where there are more agents or complex environment.

Pure Pools Comparison

We further compared our pool with two other pools: Init Pool, in which all agents act unpredictably, and Final Pool, in which all agents are well-trained. The results are shown in Fig. 5. We evaluate adaptive agents on three sets: (1) a set of random agents (Fig. 5 (1st row)), a set of trained agents (Fig. 5 (2nd row)), and a mix of different reinforcement learning agents during self-play training (Fig. 5 (3rd row)). There is a contrast between the performance of adaptive agents trained on a random pool and the trained pool because they focus on different types of collaborators. The adaptive agent trained with only pool that have unpredictable behaviour behaves worst. Our adaptive agents obtained better results when paired with trained and mixed agents, and better performance than the adaptive agent trained on a well-trained pool when paired with random agents.

6 Conclusions

In this paper, we tackle the ZSC by the generative training framework, in which a pool of agents is auto-generated in the first stage and the adaptive agent is trained to cooperate with agents in this pool to achieve the task in the second stage. We focus on the first stage and propose to use a novel way to promote the diversity of the training pool using the levels of behavioural predictability. To realise the concept, we design a ToM observer equipped with a episodic memory to monitor the pool generation process. We demonstrated that agents trained to cooperate with this training pool can better adapt to novel partner meaning they are more robust and trustworthy compared to diversity based methods.

BC0

BC1

BC2

BC3

BC4

BC5

Avg.

100

80

60

40

Acknowledgements

This research was partially funded by the Australian Government through the Australian Research Council's Discovery Project funding scheme (project DP210102798). The views expressed herein are those of authors and are not necessarily those of the Australian Government or Australian Research Council.