Toward Policy Explanations for Multi-Agent Reinforcement Learning

1 Introduction

Recent years have witnessed a growing body of research in multi-agent reinforcement learning (MARL), enabling sequential decision making for a range of exciting multi-agent applications such as cooperative AI [Dafoe et al., 2020] and autonomous driving [Kiran et al., 2021]. Generating explanations about agent decisions is crucial for improving system transparency, increasing user satisfaction, and facilitating human-agent collaboration [Kraus et al., 2020; Chakraborti et al., 2020]. However, existing works on explainable reinforcement learning (RL) mostly focus on the single-agent setting [Wells and Bednarz, 2021; Heuillet et al., 2021; Puiutta and Veith, 2020].

Generating explanations for MARL agents that interact with each other in a common environment poses significant challenges. The combinatorial nature of MARL (i.e., the joint state/action space grows exponentially with the number of agents) leads to scalability issues. Explanations should provide adequate information about agent behavior, including the interaction (e.g., cooperation) among multiple agents, for user understanding. Furthermore, explanations should avoid redundant information that may overwhelm or confuse users, thus decreasing user satisfaction and trust.

To tackle these challenges, we develop novel methods to generate two types of policy explanations for MARL: (i) policy summarization, and (ii) query-based language explanations. Our methods rely on first building an abstract representation of MARL policy as a multi-agent Markov decision process (MMDP), which can be obtained by abstracting samples observed during the MARL policy evaluation. The proposed method generates a summarization about the most probable sequence of agent behavior under a given MARL policy, by finding the most probable path through the MMDP abstraction and extracting information about agent cooperation and task sequence. The generated policy summarizations can help users to have a global view of agent decisions and support human-agent collaboration (e.g., users may adjust their workflow based on agents' task sequence).

Additionally, we developed methods for generating language explanations to answer three types of queries about agent behavior, including "When do [agents] do [actions]?", "Why don't [agents] do [actions] in [states]?", "What do [agents] do in [conditions]?" Such explanations can enable users to understand specific agent decisions, debug faulty agent behavior, and refine user mental models. Our work is inspired by the method proposed in [Hayes and Shah, 2017], which computes a minimal Boolean logic expression covering states satisfying the query criteria, and converts the Boolean expression to explanations via language templates. However, we find that a naive adaptation of this method for MARL generates explanations with redundant information and has limited scalability. Further, the generated explanations do not necessarily capture agent cooperation, which is imperative for explaining MARL. We proposed improved methods that address these limitations by leveraging MARL domain knowledge (e.g., agent cooperation requirements) to filter relevant agent states and actions.

We applied a prototype implementation of the proposed methods to three benchmark MARL domains: (i) multirobot search and rescue, (ii) multi-robot warehouse, and (iii) level-based foraging [Papoudakis et al., 2021]. Experimental results demonstrate that the proposed methods can generate policy summarizations and query-based explanations for large MARL environments with up to 19 agents.

Finally, we conducted a user study to evaluate the quality of generated explanations. We measured user performance on correctly answering questions based on explanations, to test user understanding of agent behavior. We also collected user subjective ratings on explanation goodness metrics [Hoffman et al., 2018]. The results show that the generated explanations significantly improve user performance and increase subjective ratings on various metrics including user satisfaction.

2 RelatedWork

Explaining agent decision making has recently emerged as a focus area within the explainable AI paradigm. [Chakraborti et al., 2020] provides a survey about this emerging landscape of explainable decision making. [Kraus et al., 2020] proposes Explainable Decisions in Multi-Agent Environments (xMASE) as a new research direction, emphasizing many challenges of generating multi-agent explanations, such as accounting for agent interactions and user satisfaction.

Explainable RL has been attracting increasing interest, as shown in several recent surveys [Wells and Bednarz, 2021; Heuillet et al., 2021; Puiutta and Veith, 2020]. In particular, [Wells and Bednarz, 2021] points out the lack of user studies as a major limitation across existing works. Moreover, current approaches mostly focus on the single-agent setting, while generating explanations for MARL has received scant attention so far. [Kazhdan et al., 2020] extracts MARL model as abstract argumentations, which do not explicitly consider agent cooperation on the same tasks as in our work.

The proposed methods are inspired by several prior works. [Topin and Veloso, 2019] develops Abstracted Policy Graphs (i.e., Markov chains of abstract states) for explaining singleagent RL. Our work differs in that we adopt MMDP to represent multi-agent policy abstractions. [Amir and Amir, 2018] summarizes agent behavior by extracting trajectories from agent simulations and visualized them as videos. Inspired by this idea, we created GIF animations as the baseline for evaluating policy summarizations in the user study. But we extracted the sequence of agent actions differently, focusing on the agent cooperation. [Hayes and Shah, 2017] generates RL policy descriptions to answer queries. We adapted this method to multi-agent environments and proposed significant improvements, which will be described in Section 4.

3 Policy Abstraction and Summarization

We describe how to build an abstract representation of MARL policy in Section 3.1, and present the method for generating policy summarization in Section 3.2.

3.1 Policy Abstraction

In the context of MARL, a group of N agents interact with each other in a common environment and make decisions influenced by the joint states of all agents. Agent decisions can be captured by a joint policy π : X → ∆(A), which is a function mapping the set of joint states X = {(x1 , . . . , xN )} to a probabilistic distribution over the set of joint actions A = {(a1 , . . . , aN )}, where xi (resp. ai ) denotes the state (resp. action) of agent i. Once a policy is trained, agents can act upon it in any given state. But there is a lack of a global view of the entire policy. Further, the size of the policy grows exponentially with the number of agents and state variables. To address these issues, we propose to build an abstract representation of the policy as the basis for generating explanations about agent behavior.

We use the multi-agent Markov decision process (MMDP) framework to represent MARL policy abstraction. Formally, an MMDP is a tuple (S, A, T ), where S = {(s1 , . . . , sN )} is the joint (abstract) state space, A is the joint action space, and T is the transition function. Let F be a set of Boolean predicates indicating features of the MARL domain. We denote by f(xi ) = 1 if an agent state xi satisfies a feature predicate f ∈ F. An abstract state si is then given by the satisfaction of all feature predicates f ∈ F, where each bit of the binary encoding of si ∈ N corresponds to the satisfaction of a predicate f(xi ). Thus, the choice of features affects the abstraction level and should include adequate information for explanations. In this work, we assume that users specify a set of feature predicates for a given MARL domain.

Once an MARL policy is trained, we build an MMDP during the policy evaluation stage. For each sample (x, a, x′ ), determine an MMDP transition s a-→ s′ by finding the abstract state s (resp. s′ ) corresponding to x (resp. x′ ). When policy evaluation terminates (e.g., converging to the expected reward), compute the transition probability T (s, a, s′ ) via frequency counting.

Properties. The resulting MMDP is a sound abstraction of the MARL policy because, by construction, every MMDP transition with non-zero probability corresponds to at least one sampled policy decision. The state space size |S| is bounded by O(2|F|N ), depending on the number of agents N and feature predicates |F|. In practice, a trained MARL policy may only induce a small set of reachable states.

Example 1 Figure 1(a) shows an example MARL domain where three robotic agents cooperate to complete search and rescue tasks. Rescuing the victim requires the cooperation of an unmanned aerial vehicle (UAV) and an unmanned ground vehicle (UGV). Any agent can fight the fire, which is blocked by the wall and obstacle. Removing the obstacle requires the cooperation of two UGVs. Given a trained MARL policy, we build an MMDP abstraction with 6 feature predicates indicating whether each task is detected or completed (e.g., victim detect, victim complete). An agent can only detect a task in a neighboring grid (e.g., UAV detects the victim in Figure 1(a)). The resulting MMDP has 63 (reachable) states and 577 transitions.

3.2 Policy Summarization

A policy abstraction containing hundreds of states and transitions is too complex for humans to understand. An alterna-

Algorithm 1 Generating Policy Summarization

Input: policy abstraction M = (S, A, T ), task completion predicates Fc ⊆ F Output: policy summarization Z 1: Z ← {} 2: Compute the most probable path ρ throughM 3: for 0 ≤ t ≤ |ρ| do 4: y ←new array 5: for 1 ≤ i ≤ N do 6: y[i] ← {} 7: for f ∈ Fc do 8: if agent state sit in the path ρ satisfies f then 9: insert f to y[i] 10: insert non-empty array y to Z 11: return Z

tive way of communicating agent behavior is to show execution traces; however, a lengthy trace may be burdensome for users to review. To overcome these limitations, we develop a method to generate policy summarization, illustrating the agent cooperation and task sequence for the most probable sequence of agent behavior under a given MARL policy.

Algorithm 1 shows the proposed method, which takes the input of a policy abstractionMand a set of predicates Fc representing the completion of tasks (subgoals) in a given MARL domain. The first step is to compute the most probable path ρ = s0 a0 -→ s1 a1 -→ · · · from the initial state to a goal state in the MMDP M, which represents the most probable sequence of agent decisions under the policy. This problem can be solved by converting the MMDP to a directed weighted graph with edge weight e(s, a, s′ ) = - log T (s, a, s′ ) for each transition, and then applying the Dijkstra's algorithm [Dijkstra et al., 1959] to find the shortest path.

Next, the algorithm loops through every joint state st in the path ρ to extract the agent cooperation and task sequence. At each step t, the algorithm checks if an agent state sit satisfies any task completion predicate f ∈ Fc and inserts completed task f into the array element y[i] (line 4-9). An agent only satisfies a task completion predicate at step t when it finishes the task and receives a reward. We assume that if a task is completed via the cooperation of multiple agents, they must satisfy the task predicate f at the same step t and each receive a portion of the reward. Thus, the agent cooperation is represented as multiple elements of the array y sharing the same task. Only non-empty arrays containing completed tasks are inserted into the summarization Z. When the algorithm terminates, the generated summarization is visualized as a chart, with each column corresponding to a non-empty y-array and each row representing an agent's task sequence.

Properties. The generated policy summarization Z is sound, because it is derived from the most probable path of a sound policy abstraction (see Section 3.1). The complexity of computing the most probable path is bounded by O(|S|2 ), following the complexity of the Dijkstra's algorithm and depending on the MMDP state space size. The rest of Algorithm 1 is bounded by O(|ρ| · N · |Fc|), depending on the path length and the number of agents and tasks.

Example 2 We apply Algorithm 1 using the policy abstraction and task predicates from Example 1. There are 8 states in the most probable path from the initial state (i.e., all agents starting in the green grid) to a goal state (i.e., all tasks have been completed). Figure 1(b) visualizes the generated summarization, with column names (i.e., T1, T2, T3) indicating the sequence of task completions: UGV2 and UAV cooperate to rescue the victim; next, UGV1 and UGV2 cooperate to remove the obstacle; and lastly, UAV fights the fire.

4 Query-Based Explanations

While policy summarization provides a global view of the agent behavior under a MARL policy, users may also query about specific agent decisions. In this section, we develop methods for generating language explanations to answer the following three types of queries:

• "When do [agents] do [actions]?" for identifying conditions for action(s) of a single or multiple agent(s) .
• "Why don't [agents] do [actions] in [states]?" for understanding differences in expected and observed behaviors of a single or multiple agent(s).
• "What do [agents] do in [predicates]?" for revealing agent behavior under specific conditions described by the given predicates.

Our work is inspired by a method developed in [Hayes and Shah, 2017] to generate query-based explanations for singleagent RL. In the following, we propose new methods to tackle limitations posed by adapting this baseline method to multiagent environments.

4.1 Explanations forWhen Query

Algorithm 2 presents both the baseline and proposed methods for answering "When do agents Gq do actions Aq?", where Gq and Aq are sets of agents and actions, respectively. The text in blue highlights changes about relevancy filters (RF) for the proposed method (called WithRF) compared to the baseline (called NoRF).

WithRF starts the algorithm (line 1-5) by identifying relevant agents G, features F , and action sets A based on domain knowledge (e.g., agent cooperation requirements). For example, consider a query "When does UAV rescue the victim?". The domain knowledge is that rescuing the victim requires the cooperation of a UAV and a UGV (Example 1). Thus, the relevant agent set G is {UVA, UGV 1, UGV 2}. The relevant feature set F is {victim detect, victim complete}, while predicates about the fire and obstacle are irrelevant. The relevant action sets A is an array with each element representing one possible set of agent actions required for cooperation: [{UAV rescue,UGV1 rescue}, {UAV rescue,UGV2 rescue}], which can be generated based on the aforementioned domain knowledge about agent cooperation requirements.

Both NoRF and WithRF loop through all the joint states s ∈ S of the policy abstraction MMDP and check all the enabled (i.e., with non-zero transition probability) joint actions a in state s. In line 9 of Algorithm 2, NoRF checks if a is compatible with Aq; that is, every agent action a ∈ Aq is contained in the joint action a = (a1 , . . . , aN ). By contrast, WithRF checks if a is compatible with at least one set of relevant actions contained in the array A. Following the previous example, NoRF checks if a contains UAV rescue, while WithRF checks if a contains {UAV rescue, UGV1 rescue} or

Algorithm 2 Generating Query-Based Explanations

Input: policy abstraction (S, A, T ), query "when do agents Gq do actions Aq?" Output: explanations E 1: G ← {}; F ← {}; A ← [{}] 2: for all agent action ai ∈ Aq do 3: insert all relevant agents of ai to G 4: insert all relevant features of ai to F 5: insert all relevant action sets of ai to A 6: V ← {}; ¯ V ← {} 7: for all joint state s ∈ S do 8: for all joint action a enabled in s do 9: if a is compatible with Aq [replace Aq with A] then 10: insert s to V 11: else 12: insert s to ¯ V 13: B1 ← States2Boolean(V ); B0 ← States2Boolean( ¯ V ) 14: ϕ ←Quine-McCluskey(ones=B1, zeros=B0) 15: translate ϕ to explanations E via language templates 16: return E 17: function STATE2BOOLEAN(W) 18: B ← {} 19: for all s = (s1 , . . . , sN ) ∈ W do 20: for 1 ≤ i ≤ N [replace with i ∈ G] do 21: C ← feature predicate valuations of state si 22: for f ∈ F [replace with f ∈ F ] do 23: insert C(f) to B 24: return B

{UAV rescue,UGV2 rescue}. Since each element of A is a super-set of Aq, the WithRF check is more restrictive.

If a state s has at least one enabled action a passing the aforementioned checks, s is inserted to the target states set V ; and to the non-target states set ¯ V otherwise. The intuition is that the generated explanations should describe target states satisfying the query criteria and exclude conditions of non-target states. WithRF poses further restrictions that target states need to satisfy criteria captured by relevant actions A, such as agent cooperation requirements. Thus, explanations generated by WithRF can provide information about agent cooperation, which may be missed by NoRF explanations.

Next, the algorithm converts the states set V (resp. ¯ V ) to a list of Boolean formulas B1 (resp. B0) via the function described in line 17-24. Given a joint state s = (s1 , . . . , sN ), NoRF finds valuations of every feature predicates f ∈ F for all agent state si and insert them to the list B. By contrast, WithRF only inserts to B the valuations of relevant features f ∈ F in relevant agent states si for all i ∈ G. Following the previous example, WithRF only considers Boolean formulas related to relevant feature predicates {victim detect, victim complete}, filtering out features related to the fire and obstacle.

Lastly, the algorithm supplies Boolean formulas B1 and B0 to the Quine-McCluskey algorithm [Quine, 1952] and obtains a minimized Boolean formula, which can be translated into language explanations following [Hayes and Shah, 2017]. The runtime of Quine-McCluskey grows exponentially with the number of variables. Thus, WithRF is more efficient than NoRF, due to the decreased number of Boolean variables. Moreover, filtering out irrelevant agents and features helps WithRF to prevent redundant information in the generated explanations.

Properties. Following the Quine-McCluskey, the complexity of NoRF is bounded by O(3N·|F| / ln(N · |F|)). The complexity of WithRF is reduced to O(3|G|·|F | / ln(|G| · |F |)).

Example 3 Table 1 (first row) shows the explanations generated by NoRF and WithRF for a when query. The NoRF explanation contains redundant information about the fire and obstacle that are irrelevant to the query. The WithRF explanation completely captures the required agent cooperation for the query, which is missed by the NoRF explanation.

4.2 Explanations forWhy Not Query

The query "Why don't agents Gq do actions Aq in the joint State sq?" can be answered by modifying Algorithm 2 as follows. In line 10, adding s to ¯ V instead of V . Remove line 11-12 and add a new line for inserting the query state sq to V . The modified algorithm (see Appendix A in [Boggess et al., 2022]) generates an explanation describing the differences between the observed behavior in the target query state sq and the expected behavior of states with actions compatible with the query actions Aq (NoRF) or relevant action sets A (WithRF). The complexity of the modified algorithm follows Algorithm 2.

Example 4 Table 1 (second row) shows the explanations generated by NoRF and WithRF for a why not query about the behavior of two agents UGV1 and UGV2 in the state shown in Figure 1(a). The WithRF explanation captures the required agent cooperation for removing the obstacle, while the NoRF explanation fails to provide such information.

4.3 Explanations forWhat Query

To answer the query "What do agents Gq do when satisfying predicates Fq?", we first identify all the satisfying joint states s = (s1 , . . . , sN ); that is, for all i ∈ Gq, agent state si satisfies predicates Fq. The baseline NoRF method is to generate a list of all possible enabled actions for agents Gq in these states. The proposed WithRF method improves the baseline by filtering agent actions that are relevant to predicates Fq and finding the most likely relevant agent actions from the list via frequency counting. Since the proposed methods 1 do not need to call the Quine-McCluskey, the complexity of both NoRF and WithRF are only bounded by O(|Gq| · |S| · |A|), depending on the number of query agents, joint state space, and joint action space of the policy abstraction.

Example 5 Table 1 (third row) shows the explanations generated by NoRF and WithRF for a what query. The WithRF explanation is more concise than the NoRF explanation and only contains relevant action for the query predicate.

5 Computational Experiments

We implemented and applied the proposed methods to three MARL domains. The first domain is multi-robot search and rescue (SR) similar to Example 1. The second and third domains are benchmarks taken from [Papoudakis et al., 2021]. Multi-robot warehouse (RWARE) considers multiple robotic agents cooperatively delivering requested items. Level-based foraging (LBF) considers a mixed cooperative-competitive game where agents must navigate a grid world to collect randomly scattered food. Our implementation used the Shared Experience Actor-Critic [Christianos et al., 2020] for MARL policy training and evaluation. All models were trained and evaluated to 10,000 steps, or until converging to the expected reward, whichever occurred first. The experiments were run on a laptop with a 1.4 GHz Quad-Core Intel i5 processor and 8 GB RAM.

1 See Appendix A in [Boggess et al., 2022] for the pseudos code.

Table 2 shows experimental results. For each domain, we report the number of agents N and the number of states |S| and transitions |T | of generated policy abstraction MMDP. It is unsurprising that the MMDP size grows exponentially with the number of agents. We report the most probable path length |ρ| and the chart size |Z| of generated policy summarizations, which are more compact and easier to interpret than complex MMDP abstractions. All summarizations were generated within 1 second (thus not shown in the table). Additionally, we compare NoRF and WithRF methods in terms of the number of clauses in the generated query-based explanations and runtime. The results show that WithRF is more succinct in general and has better scalability than NoRF. In particular, NoRF failed to generate explanations for "when" and "why not" queries within an hour for large cases with more than 4 agents, while WithRF generated explanations for all cases within seconds. Both methods generate explanations for "what" queries efficiently, thanks to the lower complexity than other queries (see Section 4).

In summary, experimental results demonstrate that the proposed methods can generate policy summarizations and query-based explanations for large MARL domains (e.g., RWARE with 19 agents, which is the largest number of possible agents in the provided environments).

6 User Study

We conducted a user study 2 via Qualtrics to evaluate the quality of generated explanations. We describe the study design in Section 6.1 and analyze results in Section 6.2.

2 This study was approved by University of Virginia Institutional Review Boards IRB-SBS #4701. Study details (e.g., questionnaire, interface) are included in Appendix B of [Boggess et al., 2022].

6.1 Study Design

Participants. We recruited 116 eligible participants (i.e., fluent English speakers over the age of 18) through university mailing lists. 62.1% of participants self-identified as male, 37.1% as female, and 0.8% preferred not to say. The age distribution is 76(18-24), 31(25-34), 7(35-49), 2(50-64). Participants were instructed to answer multiple-choice questions about agent behavior for multi-robot search and rescue tasks. They were incentivized with bonus payments to answer questions correctly based on the provided explanations. To ensure data quality, attention checks were injected during the study.

Independent variables. We employed a within-subject study design with the explanation generation methods as independent variables. Participants were asked to complete two trials for evaluating policy summarizations. They were presented with charts generated by Algorithm 1 in one trial, and GIF animations illustrating the most probable sequence of agent behavior (i.e., visualization of the most probable path in the policy abstraction) in the other trial. For each trial, there were two questions about agent behavior in various environments (i.e., 3×6 and 6×6 grid world). Questions used in the two trials are different but had similar difficulty. All participants were presented with the same set of four randomly generated questions for summarization trials. To counterbalance the ordering confound effect, they were randomly assigned to answer the first two questions based on either charts or GIF, and the other two questions based on the remaining method. Additionally, participants were asked to complete two trials for evaluating query-based explanations generated by NoRF and WithRF methods, with 6 questions (2 environments × 3 query types) in each trial. Participants answered the same set of 12 randomly generated questions for query-based trials, and were randomly assigned to different groups similarly to summarization trials.

Dependent measures. We measured user performance by counting the number of correctly answered questions in each trial. In addition, at the end of each trial, participants were asked to rate in a 5-point Likert scale (1 - strongly disagree,

5 - strongly agree) about explanation goodness metrics (i.e., understanding, satisfaction, detail, completeness, actionability, reliability, trust) [Hoffman et al., 2018].

Hypotheses. We make the following hypotheses in this study.

• H1: Chart-based summarizations lead to better user performance than GIF-based.
• H2: Chart-based summarizations yield higher user ratings on explanation goodness metrics than GIF-based.
• H3: Query-based explanations generated by WithRF lead to better user performance than those by NoRF.
• H4: Query-based explanations generated by WithRF yield higher user ratings on explanation goodness metrics than those by NoRF.

6.2 Results Analysis

We used a paired t-test to evaluate hypotheses H1 and H3, and used the Wilcoxon Signed-rank test to evaluate hypotheses H2 and H4. We set the significant level as α = 0.05.

Evaluating policy summarizations. Participants answered more questions correctly with chart-based summarizations (M=1.8 out of 2, SD=0.6) than GIF-based (M=0.9 out of 2, SD=0.4), with statistical significance (t(462)=-15.8, p ≤0.01, d=1.5). Thus, the data supports H1.

Figure 2 shows average participant ratings about summarizations. Chart-based summarizations yield higher ratings on the perceived completeness than GIF-based with statistical significance (W=371.5, Z=-2.4, p ≤0.02, r=-0.2). But no significant difference was found regarding the other metrics. Thus, the data partially supports H2.

Evaluating query-based explanations. Participants answered more questions correctly with explanations generated by WithRF (M=5.2 out of 6, SD=1.7) than NoRF (M=2.3 out of 6, SD=1.0), with statistical significance (t(1390)=-21.1, p ≤0.01,d=2.0). Thus, the data supports H3.

Figure 3 shows that participants gave higher average ratings to WithRF explanations than NoRF explanations. The Wilcoxon test found significant differences on all metrics: understanding (W=319.5, Z=-4.9, p ≤0.01, r=-0.3), satisfaction (W=266.0, Z=-7.0, p ≤0.01, r=-0.5), detail (W=484.0, Z=-3.7, p ≤0.01, r=-0.2), completeness (W=494.5, Z=-6.4, p ≤0.01, r=-0.4), actionability (W=167.0, Z=-6.9, p ≤0.01, r=-0.5), reliability (W=382.5, Z=-3.6, p ≤0.01, r=-0.2), and trust (W=217.0, Z=-3.4, p ≤0.01, r=-0.2). Thus, the data supports H4.

Figure 3: Mean and SD of participant ratings about query-based explanations ("*" indicates statistically significant difference).

Discussion. In summary, the data supports all hypotheses, while H2 is only partially supported because the statistical test found no significant differences between chart-based and GIF-based summarizations on most metrics. However, Figure 2 shows that participants rated chart-based summarizations close to 4 (agree) on all metrics, and above GIF-based ratings on all metrics except understanding, reliability, and trust. This may be because users showed a strong preference toward the moving nature of GIF animations and the visualized effects of agents completing tasks. But watching a GIF can be more time-consuming and less informative than a quick glance at the chart. This is supported by the results that participants were able to answer more questions correctly with chart-based summarizations, and they rated this method significantly higher on completeness (i.e., providing needed information). Meanwhile, query-based explanations generated by the proposed WithRF method led to significantly better user performance and higher user ratings on all metrics, because users prefer succinct WithRF explanations with adequate information about agent behavior and cooperation. By contrast, NoRF explanations do not necessarily provide essential information about agent cooperation for correctly answering questions, and may contain redundant information that decreases user satisfaction.

7 Conclusion

In this work, we developed methods to generate policy summarizations and query-based explanations for MARL. Experimental results on three MARL domains demonstrate the scalability of our methods. Evaluation via a user study shows that our generated MARL policy explanations can improve user understanding about agent behavior and enable them to answer more questions correctly, while maintaining very positive ratings on explanation goodness metrics.

The proposed methods are independent from the inner working of MARL methods, only relying on policy abstractions that can be built via observing samples during the policy evaluation. Although we only used one MARL method in our experiments, the proposed methods can be used to generate policy explanations for different MARL methods. As part of the future work, we plan to apply the proposed methods to a wide range of MARL methods and domains.

Acknowledgments

This work was supported in part by U.S. National Science Foundation grant CCF-1942836, U.S. Office of Naval Research grant N00014-18-1-2829, Israel Science Foundation grant 1958/20, EU Project TAILOR grant 992215, and by the Data Science Institute at Bar-Ilan University. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the grant sponsors.