Variational Offline Multi-agent Skill Discovery

1 Introduction

Skill discovery aims at extracting useful temporal abstractions from decision-making sequences. The downstream policy learning can be much more efficient by simply composing the discovered skills temporally into complex maneuvers. Also, skills can potentially be transferred among tasks to facilitate multi-task learning. Despite considerable research on single-agent skill discovery [Eysenbach et al., 2019; Chen et al., 2023a], skill discovery in multi-agent reinforcement learning (MARL) remains under-explored. A straightforward approach is to discover single-agent skills for each agent independently and then learning a multi-agent meta policy to coordinate their use, as in [Yang et al., 2020;

Sachdeva et al., 2021]. However, multi-agent coordination can not be abstracted in such individual skills. On the other hand, there are a limited number of works [Chen et al., 2022; Yang et al., 2023] on discovering skills for the entire team of agents. However, in multi-agent tasks, coordination patterns can emerge within subgroups of varying scales (from 1 to n), and team skills (i.e., n-agent skills) can be inflexible to use.

Notably, complex multi-agent tasks can often be decomposed as a series of subtasks, each of which requires participation of a subgroup of agents for a certain duration. The policies for these subgroups can be abstracted as multi-agent skills. While agents can explore various forms of collaboration in an online setting (by interacting with the environment), offline multi-agent skill discovery in contrast must infer latent coordination patterns from agent interactions in the offline data, with the subgroup size arbitrarily varying from 1 to n. This gives rise to a combinatorial problem of dynamic subgroup division and forming temporal abstractions within each subgroup for skill discovery, which is a significant new challenge. To the best of our knowledge, this is the first work to fully automate the extraction of collaborative patterns among agents as subgroup skills from offline data. We also note that the problem is different from (online) role-based MARL [Xu et al., 2023; Zhou et al., 2024]. They instead focus on partitioning agents into subdivisions that consist of agents with similar responsibilities (i.e., roles), sharing the same policy and thus homogeneous behaviors. Our goal is to learn multi-agent skills - a collective set of single-agent skills taken by a subgroup where agents could have distinct yet coordinated behaviors.

To be specific, we provide effective auto-encoder frameworks for extracting embeddings of subgroup coordination patterns from offline data as a codebook, where each code corresponds to a multi-agent skill and provides abstractions in both subgroup- and temporal-level. We propose two scheme designs for this purpose: VO-MASD-3D and VO-MASDHier. In VO-MASD-3D, three-dimensional codebooks are adopted, where each multi-agent skill code consists of several single-agent skill codes such that it can be used to represent subgroup behaviors. In contrast, VO-MASD-Hier employs a two-level codebook: the bottom codes encode individual behaviors, while each top code is aggregated from a set of bottom codes to encode subgroup behaviors. Further, to enable automatic grouping while forming temporal abstractions, we co-train a grouping function with the proposed auto-encoder schemes. Using this function, agents can be dynamically grouped based on the environment state, and each subgroup can then be assigned a multi-agent skill of the corresponding size. Importantly, our algorithm is designed to work with multi-task data, such that the discovered skills can be utilized in multiple relevant tasks (without retraining). Empirical results on challenging StarCraft tasks [Samvelyan et al., 2019; Ellis et al., 2024] demonstrate the superiority of the discovered multi-agent skills using our algorithm even in previously unseen tasks, and show the great advantages brought by the use of skills in long-horizon multi-agent tasks characterized by sparse reward signals.

2 Background

Dec-POMDP. This work focuses on a fully cooperative multi-agent setting with only partial observation for each agent, which can be modeled as a decentralized partially observable markov decision process (Dec-POMDP) [Oliehoek et al., 2016] and described with a tuple G = ⟨n, I, S,O, F,A, µ, P,R, γ⟩. At a time step, each agent i ∈ I = {1, · · · , n} would obtain a local observation oi ∈ O from the observation function F(s, i) : S × I → O, where s is the real state of the environment, and determine its action ai ∈ A. This would lead to a state transition in the environment according to the function P(s′ | s,⃗a) : S × An × S → [0, 1] and all agents would receive a shared team reward r = R(s,⃗a) : S × An → R. To mitigate the issue of partial observability, each agent i holds an actionobservation history τ i t-1 = (o1, a1, · · · , ot-1, at-1) and decides on its action ait based on a policy πi (ait | oit , τ i t-1 ). The goal of MARL in a Dec-POMDP can be formally defined as max⃗π Eµ,⃗π,P,R [∑∞t=0 γ t rt], where ⃗π = (π1 , · · · , πn ) and µ(s0) : S → [0, 1] denotes the distribution of the initial state.

CTDE. The paradigm of centralized training with decentralized execution (CTDE) [Oliehoek et al., 2008] is proposed for solving Dec-POMDP and has gained substantial attention. In this paradigm, agents learn their policies with access to global information (e.g., the state s) during centralized training and only rely on their local action-observation histories for decentralized execution. Notably, the discovered skills with our algorithm can be easily integrated into the CTDE paradigm. We select MAPPO [Yu et al., 2022b] as the base CTDE MARL algorithm throughout this work, as it has shown superior performance across various MARL benchmarks. It learns a decentralized actor π(ait | oit , τ i t-1 ), which is used for each agent i ∈ I to determine its action ait based on its individual action-observation history (oit , τ i t-1 ), and a centralized critic V (st). Viewing the n agents as a whole, the critic function is trained as in a single-agent RL algorithm - PPO [Schulman et al., 2017], while the actor (of each agent) is trained to maximize the advantage function defined with the team reward and centralized critic.

Skill & Task Decomposition. In single-agent scenarios, skills are used as temporal abstractions of an agent's behaviors. This is inspired by the fact that complex tasks can usually be decomposed as a sequence of subtasks and each subtask can be handled with a corresponding subpolicy, i.e., a skill. With skills, an agent learns a hierarchical policy, where the low-level part πl(a | s, z) is the skill policy and the highlevel part πh(z | s) determines the skill selection. Each skill z ∈ Ωz , after being selected, will be executed for H time steps - a predefined subtask duration. However, in multiagent scenarios, task decomposition occurs not just at the temporal level but also at the agent level, since the overall multi-agent task can be viewed as several subgroup tasks executed in parallel. Here, we define such a task decomposition:

Definition 1. Given a cooperative multi-agent task ⟨n, I, S,O, F,A, µ, P,R, γ⟩, at a time step, it can be decomposed into a set of m subtasks, each of which is solved by a subgroup of agents for H time steps and can be represented as a tuple ⟨nj , Ij , S, O, F, A, µ, P, Rj , γ⟩. Here, ∑mj=1 nj = n, ∪jIj = I, and Ij ∩ Ik = ∅ (∀ j ̸= k).

Certain subtasks may frequently occur, such as passing and cutting cooperation among two or three players in a football match, and their subpolicies, showing coordination patterns, can be extracted as multi-agent skills and transferred across similar tasks for reuse. In this work, we propose an algorithm for discovering such multi-agent skills Z ∈ ΩZ from multiagent interaction data.

Related Works. In Appendix A of an extended version of this paper 1 , we provide a thorough review of research on applying skills in MARL. There are three main categories: MARL with single-agent skills [Lee et al., 2020; Yang et al., 2020; Chen et al., 2023a; Chen et al., 2023c], role-based MARL [Yang et al., 2022; Xu et al., 2023], and team skill discovery [Chen et al., 2022; Chen et al., 2023b]. As a summary, research on multi-agent skill discovery is still at an early stage of development, especially for the offline setting. Even without prelearned skills, when dealing with a complex multi-agent task, the agents would implicitly learn to decompose the overall task into several subtasks, assign a subgroup for each subtask, and develop a joint policy (i.e., multi-agent skill) within the subgroup to handle the corresponding subtask. Replacing primitive actions with singleagent skills or role policies could make such a learning process more efficient, as agents can assemble these higher-level abstractions to obtain the required subgroup policies more easily. As the first offline multi-agent skill discovery algorithm, our work takes one step further by directly identifying subgroups, which could change throughout a decision horizon, and extracting their coordination patterns as multi-agent skills. With these joint skills, the MARL process could be greatly simplified, since agents only need to select correct multi-agent skills without considering grouping with others or forming subgroup policies. Thus, multi-agent skills represent a more efficient form of knowledge discovery.

3 Proposed Approach

Variational Offline Multi-agent Skill Discovery (VO-MASD) aims to extract a finite set of multi-agent skills from given offline trajectories. Proposed for Computer Vision, VQ-VAE [van den Oord et al., 2017; Chen et al., 2024] provides a fundamental manner to learn discrete representations for complex, high-dimensional data. Besides the encoder and decoder as used in VAEs [Kingma and Welling, 2014], a codebook containing a finite set of codes, each of which is a latent representation of the data, is learned. In this case, VQ-VAE is a natural choice for skill discovery, with each code working as a skill embedding Z. Each Z would correspond to a skill policy πl(⃗a | ⃗τ,Z) that leads to continuous multiagent behaviors. In this section, we present two schemes of VO-MASD based on VQ-VAE by adopting novel codebook designs and involving an automatic grouping module. The challenge is to extract temporal-level abstractions (i.e., useful control sequences) and agent-level abstractions (i.e., multi-agent coordination) at the same time, without using domain knowledge or task-specific reward signals. In this way, VO-MASD can be applied to a mixture of multi-task data and the learned skills are generalizable to relevant tasks.

1 https://arxiv.org/abs/2405.16386

3.1 VO-MASD Based on 3DCodebooks

VQ-VAE typically adopts a 2D codebook [e1, · · · , ek] ∈ Rk×d , where ei ∈ R1×d is a latent representation. However, in our case, there are three levels of abstractions: primitive actions → single-agent skills → multi-agent skills. As part of our novelty, we propose to use 3D codebooks within Rk×m×d to represent a set of (i.e., k) m-agent skills. Each code ei = [ei,1, · · · , ei,m] ∈ Rm×d represents a multi-agent skill composed of m single-agent skills. Note that m ranges from 1 to n (i.e., the size of the team).

A straightforward approach to utilize such codebook design for skill discovery is repeatedly applying a VQ-VAE with an m-agent codebook to m-agent skill discovery, for m = 1, · · · , n. As some variational methods for single-agent skill discovery [Campos et al., 2020; Ajay et al., 2021], the objective for learning m-agent skills could be minimizing the reconstruction error of m-agent trajectory segments. Ideally, after training, each code can represent a coordination pattern among m agents and the code-conditioned decoder can be used as an m-agent skill policy. However, if there are no coordination involving m agents in the offline data, the effort to discover m-agent skills would be wasted. Also, in this way, the learning processes for skills involving different numbers of agents are independent and cannot benefit from each other.

In this case, we introduce a grouping function hψ that dynamically groups agents throughout an episode to identify existing coordination patterns in the offline data and unify the training of skills with different numbers of agents. The skill discovery process is illustrated as Figure 1. As shown in (a), at time step t, for each agent i, we encode its following H time steps, i.e., τ i = [oit , ait , · · · , oit+H-1 , ait+H-1 ], into a skill embeddings zie using the encoder fθ. Also, each agent i selects its group based on the global state st and group choices of previous agents g1:i-1 using a grouping function hψ. There can be at most n groups, when all agents choose to use individual skills. Notably, both hψ and fθ are shared by all agents. Subsequently, in (b), the skill embeddings z1:n e from the encoder are first clustered based on the grouping result g1:n : if m agents choose the same group (indicated by the one-hot output gi ), they aim to form an m-agent coordination skill and their respective embeddings will be concatenated in the sequence of their agent indices, resulting in an m × d joint embedding zj1:m e . Then, as in VQ-VAE, the code that is the closest to zj1:m e in the m-agent codebook (i.e., zj1:m q ) is queried to work as the skill code. Finally, in (c), a decoder πϕ maps the skill code back to an m-agent trajectory segment, i.e., ˆ τ j1:m . The training objective for subgroup j1:m is:

As shown in Figure 1, zji e = fθ(τ ji ), ej1:m = arg mine∈Em ∥zj1:m e - e∥2 (Em denotes the m-agent codebook), and zji q = eji . L3D(τ j1:m ) is an objective with respect to (w.r.t.) θ, ϕ,Em. As in VQ-VAE, the first term in Eq (1) is a reconstruction loss of trajectory segments, and the last two terms move the skill codes (e.g., eji ) and encoder embeddings (e.g., zji e ) towards each other, where sg represents the stop gradient operator. Through reconstructing m-agent (m ∈ {1, · · · , n}) trajectory segments in an autoencoder framework, representations of m-agent skills can be extracted as codes in the codebook. The overall objective for VO-MASD-3D is as below:

Here, DH is a (multi-task) offline dataset with trajectories segmented every H time steps; each n-agent trajectory segment is partitioned into subgroups (e.g., j1:m) based on the grouping function hψ. Note that, unlike fθ, E1:n, and πϕ, hψ cannot be trained in an end-to-end manner by minimizing Eq (2), since its output g1:n are used for clustering which is not an differentiable operation. Thus, we choose to optimize hψ with MAPPO, where each agent i takes an action gi to maximize the global return -L3D . In this way, all modules in Figure 1 are effectively updated with a common objective.

Algorithm 1 MAPPO with learned skills

Input: πω, Vη, πϕ, hψ, E1:n, Env Initialize πω, Vη while not converged do Buffer ← ∅ for b = 1 · · · B do Initialize τ 1:n -H , Traj ← ∅, ˜ r ← 0 for t = 0 · · · T do if t%H == 0 then zi t , τ i t ← πω(oit , τ i t-H ), i = 1 · · · n Get e1:n based on z1:n t using hψ and E1:n, following Fig 1 (b) Add (˜ r, st, o1:n t , τ 1:n t-H , z1:n t ) to Traj ˜ r ← 0 end if ait ← πϕ(oit |ei ), i = 1 · · · n rt, st+1, o1:n t+1 ← Env(a1:n t ), ˜ r += rt end for Buffer ← Buffer ∪ Traj end for Train πω, Vη based on Buffer using MAPPO end while

This framework is advantageous: (1) the training of skills with different number of agents can facilitate each other, as they share all modules but the codebook; (2) the modeling of temporal- and agent-level abstractions within multi-agent skills are decoupled as training the decoder to reconstruct single-agent trajectories and training the grouper for automatic grouping; (3) the grouper is trained to form subgroups only when it enhances the overall pattern extraction objective (i.e., Eq (2)), ensuring that each subgroup, along with its policy, corresponds to a genuine coordination pattern.

We illustrate how to utilize the discovered skills in CTDE MARL in Figure 2. Also, Alg 1 outlines the detailed training process for a decentralized actor πω and centralized critic Vη using MAPPO in a multi-agent task Env, leveraging the pretrained components hψ, E1:n, and πϕ. In particular, every H time steps, the actor produces a skill embedding zi ∈ R1×d for each agent i. z1:n are mapped to the closest multi-agent skill codes e1:n using the grouper hψ and codebook E1:n, following Figure 1 (b). Then, for the next H time steps, each agent i interacts with Env using corresponding πϕ(ai | si , ei ), i.e., the decoder working as the skill policy. Based on the interaction transitions, i.e., {(st, o1:n t , τ 1:n t-H , z1:n , ˜ rt, st+H)},

πω and Vη can be trained with MAPPO, where τ 1:n t-H are the skill - observation (i.e., z - o) history, z1:n can be viewed as (high-level) actions, and ˜ rt = ∑t+H-1 l=t rl is the skill reward.

Besides the approach illustrated in Figure 2, we propose two alternative methods for mapping z1:n to e1:n in Appendix B. All the three methods assign each multi-agent (m × d) code as a complete unit to a corresponding-size (m-agent) subgroup, such that the collaboration pattern encoded in the multi-agent code can be utilized. Alternatively, each (m× d) code can be decomposed into a set of (m) single-agent codes. Each agent i could then independently select its skill from the set of single-agent codes. In particular, the single-agent skill code closest to the agent's actor output zi would be selected. We denote this algorithm as 'VO-MASD-Mixed'. Later, we empirically compare these four skill assignment manners. Importantly, no matter which manner we choose, we only need to train a decentralized actor and a centralized critic for (online) MARL, without any additional learning effort beyond standard CTDE MARL approaches.

3.2 VO-MASD Based on a Hierarchical Codebook

Here, we present VO-MASD-Hier, which is an alternative to VO-MASD-3D and adopts a hierarchical codebook as in [Razavi et al., 2019]. Although [Razavi et al., 2019] is originally proposed for image generation, its top and bottom codebooks perfectly echo the two-level structure of multi-agent and single-agent skill embeddings. Thus, we propose to learn top and bottom codebooks as agent- and temporal-level abstractions, respectively, for multi-agent skill discovery. The overall framework of VO-MASD-Hier is shown as Figure 3. It contains a two-level codebook, i.e., Etop,Ebtm. VOMASD-Hier does not need to learn n codebooks (i.e., E1:n) as in VO-MASD-3D, while VO-MASD-3D can potentially make better use of domain knowledge. For example, if the subgroup scale (e.g., m) is known in advance, VO-MASD-3D only needs to learn E1 and Em, while VO-MASD-Hier cannot specify the number of agents within a multi-agent skill.

The skill discovery process is detailed as follows. In Figure 3 (a), the embedding process of each individual trajectory segment τ i (i = 1, · · · , n) is the same as the one of VO-MASD-3D (i.e., Figure 1 (a)). Subsequently, in (b), the skill embeddings z1:n btm are clustered based on the output from the grouping function, i.e., g1:n , and then embeddings within the same subgroup (e.g., z11:2 btm ) are aggregated to a higherlevel representation (e.g., z1 top ) which is then used to query a top code (e.g., q1 top ). Note that the aggregator fθtop uses a multi-head attention module [Vaswani et al., 2017] to process varied-length inputs and so can be shared by all subgroups (of varied sizes). In Figure 3 (c), for each agent i, a bottom code qibtm is assigned based on its individual skill embedding zibtm . Finally, qibtm and qli top, involving temporal- and agent-level abstractions respectively, are used to decode/reconstruct τ i . The overall objective is minθtop,btm,Etop,btm,πϕ Eτ 1:n∼DH LHier (τ 1:n ):

This loss function is similar with Eq (1), i.e., to reconstruct the input multi-agent trajectory segment, and move the codes and corresponding skill embeddings towards each other.

As for the design intuition, considering the first term in Eq (3), the gradient w.r.t. the bottom code qibtm only comes from reconstructing agent i's individual skill trajectory τ i . However, the gradient w.r.t. the top code qli top is derived from reconstructing the multi-agent skill trajectories of the subgroup li that i belongs to, since each agent j in li would adopt qli top as the decoder condition to reconstruct corresponding τ j . This reflects that the top and bottom codebooks are trained to embed subgroup- and temporal-level abstractions, respectively. Notably, both VO-MASD-3D and VO-MASD-Hier follow the inductive bias: primitive actions → single-agent skills → multi-agent skills. That is, each single-agent skill code is trained to embed an individual trajectory and each multi-agent skill code is a composition of single-agent ones. In VO-MASD-3D, each (m× d) multi-agent code contains a set (m) of (1 × d) single-agent codes; while for VO-MASDHier, each multi-agent embedding ztop is obtained through aggregating individual skill embeddings zbtm from the same subgroup using an attention mechanism, as in Figure 3 (b).

To utilize the discovered skills in downstream online MARL, Alg 1 can be applied to VO-MASD-Hier by replacing the process in Figure 1 (b)(c) with corresponding ones in Figure 3 (b)(c). Specifically, a decentralized actor πω gives out skill embeddings z1:n btm every H time steps. hψ, Etop,btm, fθtop , and πϕ are fixed during online MARL, transforming z1:n btm to multi-agent and single-agent skill codes, i.e., q1:n top and q1:n btm .

The decoder is then used to produce skill trajectories of length H, according to πϕ(ait | oit , qli top, qibtm).

4 Evaluation andMain Results

Experiments are conducted on the StarCraft multi-agent challenge (SMAC) [Samvelyan et al., 2019] - a commonly-used benchmark for cooperative MARL. Following ODIS [Zhang et al., 2023], we adopt two SMAC task sets to test the discovered multi-task multi-agent skills. In each task set, agents control some units like marines, medivacs, and marauders, but the number of controllable agents or enemies varies across tasks in a task set. We refer to the two task sets as 'marine' and 'MMMs', which evaluate algorithm performance in scenarios with homogeneous and heterogeneous agents, respectively, and are detailed in Appendix C. For each task set, we discover skills from offline trajectories of source tasks, and then apply these skills to each task in the task set (including source and unseen tasks) for online MARL. The offline trajectories are collected with well-trained MAPPO agents, which can be viewed as expert demonstrations for the source tasks, and are included in our released code folder.

Next, we show evaluation results on several aspects. (1) In Section 4.1, we compare skills discovered using different algorithms on two SMAC task sets, based on their utility for downstream online MARL, to demonstrate the superiority of the multi-agent skills discovered by our methods. (2) In Section 4.2, we test the algorithms on a task set from another benchmark - SMACv2 [Ellis et al., 2024], which features stochastic environments. (3) In Section 4.3, we show that, for MARL tasks with sparse reward signals, hierarchical learning with skills discovered using our methods can significantly outperform usual MARL algorithms. Notably, the skills are from relevant but different tasks. (4) In Appendix F of the extended version, we provide visualizations of the skills, evaluate our algorithms on offline datasets of varying quality, and present an ablation study to support our design choices.

4.1 Utility of Discovered Skills for Online MARL

The first group of results is shown as Figure 4, where '3d', 'hier', 'mixed', 'single', and 'odis' refer to VO-MASD-3D, VO-MASD-Hier, VO-MASD-Mixed, VO-MASD-Single, and ODIS, respectively. As mentioned in Appendix A, ODIS is the only existing algorithm for discovering multi-agent temporal abstractions from offline multi-task data, and is a representative of role-based MARL. Notably, ODIS has demonstrated superior performance compared to direct imitation learning from the offline dataset, MADT [Meng et al., 2021] (an offline MARL algorithm using pretraining), and UPDeT [Hu et al., 2021] (a SOTA multi-task MARL method), making it a strong baseline for comparison 2 . VOMASD-Single represents the other main branch of hierarchical MARL - learning a set of single-agent skills and collaboratively utilizing them for MARL, which is realized through removing Etop and fθtop in VO-MASD-Hier (i.e., Figure 3). VO-MASD-Single discovers and utilizes single-agent skills, while VO-MASD-Mixed discovers multi-agent skills as in VO-MASD-3D but employs the learned skills as single-agent ones, which is detailed in the last paragraph of Section 3.1. Thus, the baselines include SOTA algorithms in this field and two variations of our algorithms to respectively show the effect of discovering and utilizing skills as multi-agent units.

Skills (of length 5) discovered from source tasks are applied to both source and unseen tasks for online MARL using Alg 1. In marine, 3m and 5m are source tasks; while in MMMs, MMM is the source task. We believe that the learning performance on unseen tasks with higher-complexity is the best way to testify the utility and generality of skills discovered with different algorithms. In particular, we track the change of win rates as the number of training samples increases, presenting the mean and 95% confidence intervals as solid lines and shaded areas, respectively. Several conclusions can be drawn from Figure 4. (1) ODIS and VO-MASDSingle, which represent two main existing approaches of applying skills in MARL, exhibit inferior performance compared to the others, especially in unseen tasks. This underscores the importance of discovering coordination patterns as multi-agent skills, which can significantly enhance performance and generality in new multi-agent tasks. (2) In marine tasks, the performances of VO-MASD-3D and VOMASD-Hier are comparable, with VO-MASD-Hier performing better in 10m and VO-MASD-3D excelling in the others. However, VO-MASD-3D's performance deteriorates in MMMs, suggesting that its design may not be well-suited for heterogeneous-agent tasks like MMM and MMM2 and indicating a potential future research direction for improvement. (3) VO-MASD-Mixed follows the same skill discovery process as VO-MASD-3D but adopts the skills as single-agent ones. Surprisingly, VO-MASD-Mixed consistently outperforms VO-MASD-3D. While VO-MASD-3D utilizes fixed combinations of single-agent (1×d) codes from the discovery stage, VO-MASD-Mixed explores all possible combinations of these (1 × d) codes to achieve a higher return, which explains its better performance. However, in the most challenging settings (i.e., 10m and MMM2), VO-MASD-Hier demonstrates better results, showing the potential benefit of utilizing discovered multi-agent skills as complete units. (4) The evaluation on MMM2 - a super-hard task setting [Samvelyan et al., 2019], demonstrates the superiority of VO-MASD-Hier over other algorithms. All algorithms, except for VO-MASDHier, exhibit large variance across different runs.

2 In ODIS, the discovered skills are used for offline MARL. For fair comparisons, we instead integrate skills from ODIS with online MARL, as in VO-MASD-3D and VO-MASD-Hier.

4.2 Evaluation Results on SMACv2

SMACv2 [Ellis et al., 2024] uses procedural content generation [Risi and Togelius, 2020] to address SMAC's lack of stochasticity. In SMACv2, for each episode, team compositions and agent start positions would be generated randomly. Thus, it is no longer sufficient for agents to repeat a fixed action sequence, but they must learn to coordinate across a diverse range of scenarios. Specifically, we select the Terran task, which involves teams composed of three types of units: marines, marauders, and medivacs. As in SMAC tasks, we train an MAPPO policy as the offline data collector. However, the learned policy can only achieve a win rate around 55% on the source tasks upon convergence, highlighting the difficulty of SMACv2. For different tasks in this task set, we vary the team size, selecting the less challenging Terran-3 and Terran-5 as source tasks and Terran-7 as the target task.

Our methods consistently outperform ODIS, discovering more effective skills for downstream MARL. We also adopt HMASD [Yang et al., 2023] as a baseline. HMASD is a SOTA online hierarchical MARL algorithm that discovers skills while forming a hierarchical policy for a specific task. Notably, for Terran-7, the skills used by our algorithms are discovered from Terran-3 and Terran-5 and remain fixed during hierarchical policy learning, whereas HMASD develops specific skills for Terran-7. Despite this, our algorithm still achieves superior performance. As discussed in Appendix A, HMASD discovers only single-agent and team skills, rather than multi-agent skills for subgroups of varying sizes. Team skills can be less flexible to use, particularly when the team composition randomly changes (as in SMACv2).

4.3 Performance inMARL with Sparse Rewards

With pretrained skills, only a high-level policy πω for skill selection needs to be trained for downstream task learning, as detailed in Alg 1, and the decision horizon of πω is reduced to the original one divided by the skill length H. Thus, learning with skills (i.e., hierarchical learning) is particularly advantageous for long-horizon tasks with sparse and delayed reward signals. To testify this, we modify the reward setups of the unseen tasks: 7m, 10m, MMM2, to be sparse, where agents receive a reward of 20 only upon eliminating all enemies; otherwise, they receive a reward 0. These three tasks, with maximum episode horizons of 110, 120, and 180 respectively, are particularly challenging.

We apply two online MARL algorithms: MAPPO [Yu et al., 2022a] and QMIX [Rashid et al., 2018], to these tasks, and they consistently fail with all-zero win rates. Although they have been proposed for years, MAPPO and QMIX remain the most robust algorithms in online MARL, as verified by extensive empirical studies [Yu et al., 2022a; Hu et al., 2023]. In contrast, with skills discovered using our algorithms: VO-MASD-3D, VO-MASD-Mixed, VOMASD-Hier, the performance can be greatly improved, as shown in Figure 6. Note that (1) skills are discovered from source tasks (rather than 7m, 10m, or MMM2) and (2) only sparse rewards are adopted for downstream online MARL. This highlights the effectiveness of hierarchical MARL when employing the multi-agent, multi-task skills discovered by our algorithms. As in Figure 4, VO-MASD-Hier achieves the best overall performance, followed by VO-MASD-Mixed. Further, we compare our methods with HMASD, which discovers skills through interaction with the environment and has proven effective in sparse reward settings. However, HMASD fails in all three tasks, highlighting the superiority of the skills learned with our methods, even though they are discovered from offline data of different tasks.

5 Conclusion

In this work, we propose novel algorithms for discovering coordination patterns among agents as multi-agent skills from offline multi-task data. The key challenge lies in abstracting agents' behaviors at both the temporal and agent levels in a fully automatic manner. We address this challenge by developing novel encoder-decoder architectures and co-training the encoder-decoder with a grouping function that dynamically groups agents. Empirical results demonstrate that multiagent skills discovered using our methods significantly enhance learning in downstream MARL tasks, particularly in scenarios with sparse reward signals.

Ethical Statement

There are no ethical issues.