Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps (Extended Abstract)

Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps (Extended Abstract)

Tobias Huber, Katharina Weitz, Elisabeth André, Ofra Amir

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
Journal Track. Pages 5747-5751. https://doi.org/10.24963/ijcai.2022/803

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as they act in large state spaces, and their decision-making can be affected by delayed rewards. In this paper, we explore a combination of explanations that attempt to convey the global behavior of the agent and local explanations which provide information regarding the agent's decision-making in a particular state. Specifically, we augment strategy summaries that demonstrate the agent's actions in a range of states with saliency maps highlighting the information it attends to. Our user study shows that intelligently choosing what states to include in the summary (global information) results in an improved analysis of the agents. We find mixed results with respect to augmenting summaries with saliency maps (local information).
Keywords:
Machine Learning: Explainable/Interpretable Machine Learning
AI Ethics, Trust, Fairness: Explainability and Interpretability
Machine Learning: Deep Reinforcement Learning
AI Ethics, Trust, Fairness: Trustworthy AI
Computer Vision: Interpretability and Transparency