Universal Reinforcement Learning Algorithms: Survey and Experiments

Universal Reinforcement Learning Algorithms: Survey and Experiments

John Aslanides, Jan Leike, Marcus Hutter

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Main track. Pages 1403-1410. https://doi.org/10.24963/ijcai.2017/194

Many state-of-the-art reinforcement learning (RL) algorithms typically assume that the environment is an ergodic Markov Decision Process (MDP). In contrast, the field of universal reinforcement learning (URL) is concerned with algorithms that make as few assumptions as possible about the environment. The universal Bayesian agent AIXI and a family of related URL algorithms have been developed in this setting. While numerous theoretical optimality results have been proven for these agents, there has been no empirical investigation of their behavior to date. We present a short and accessible survey of these URL algorithms under a unified notation and framework, along with results of some experiments that qualitatively illustrate some properties of the resulting policies, and their relative performance on partially-observable gridworld environments. We also present an open- source reference implementation of the algorithms which we hope will facilitate further understanding of, and experimentation with, these ideas.
Keywords:
Machine Learning: Reinforcement Learning
Agent-based and Multi-agent Systems: Agent Theories and Models
Uncertainty in AI: Sequential Decision Making