Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment
Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment
Wilka Carvalho, Anthony Liang, Kimin Lee, Sungryull Sohn, Honglak Lee, Richard Lewis, Satinder Singh
Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence
Main Track. Pages 2219-2226.
https://doi.org/10.24963/ijcai.2021/306
Learning how to execute complex tasks involving multiple objects in a 3D world is challenging when there is no ground-truth information about the objects or any demonstration to learn from.
When an agent only receives a signal from task-completion, this makes it challenging to learn the object-representations which support learning the correct object-interactions needed to complete the task.
In this work, we formulate learning an attentive object dynamics model as a classification problem, using random object-images to define incorrect labels for our object-dynamics model.
We show empirically that this enables object-representation learning that captures an object's category (is it a toaster?), its properties (is it on?), and object-relations (is something inside of it?).
With this, our core learner (a relational RL agent) receives the dense training signal it needs to rapidly learn object-interaction tasks.
We demonstrate results in the 3D AI2Thor simulated kitchen environment with a range of challenging food preparation tasks.
We compare our method's performance to several related approaches and against the performance of an oracle: an agent that is supplied with ground-truth information about objects in the scene.
We find that our agent achieves performance closest to the oracle in terms of both learning speed and maximum success rate.
Keywords:
Machine Learning: Deep Reinforcement Learning
Machine Learning: Deep Learning
Machine Learning: Reinforcement Learning
Machine Learning: Relational Learning