Solving Continuous Control with Episodic Memory

Solving Continuous Control with Episodic Memory

Igor Kuznetsov, Andrey Filchenkov

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence
Main Track. Pages 2651-2657. https://doi.org/10.24963/ijcai.2021/365

Episodic memory lets reinforcement learning algorithms remember and exploit promising experience from the past to improve agent performance. Previous works on memory mechanisms show benefits of using episodic-based data structures for discrete action problems in terms of sample-efficiency. The application of episodic memory for continuous control with a large action space is not trivial. Our study aims to answer the question: can episodic memory be used to improve agent's performance in continuous control? Our proposed algorithm combines episodic memory with Actor-Critic architecture by modifying critic's objective. We further improve performance by introducing episodic-based replay buffer prioritization. We evaluate our algorithm on OpenAI gym domains and show greater sample-efficiency compared with the state-of-the art model-free off-policy algorithms.
Keywords:
Machine Learning: Deep Reinforcement Learning
Machine Learning: Reinforcement Learning