Latent State Models for Meta-Reinforcement Learning
- Nagabandi, Anusha; Rakelly, Kate*; Zhao, Zihao; Finn, Chelsea; Levine, Sergey
- Accepted abstract
[Join poster session]
Poster session from 15:00 to 16:00 EAT and from 20:45 to 21:45 EAT
Obtain the zoom password from ICLR
Robots operating in the real world must be able to efficiently acquire new skills as their environment and responsibilities change, as opposed to repeatedly executing a single task. Deep reinforcement learning (RL) algorithms can be used to learn robotic skills via end-to-end training, mapping high-dimensional sensory data directly to robot actions; however, these algorithms are often impractically sample inefficient to run in the real world. Fortunately, their efficiency can be improved through the use of unsupervised learning to extract latent state representations from the inputs. Even with this improved efficiency, robots operating in the ever-changing settings of the real world could not feasibly learn each task from scratch. To this end, meta-reinforcement learning algorithms offer a promising approach for leveraging relevant previous experiences to quickly acquire new skills. Once trained, new skills can be acquired rapidly; however, the onerous data requirements of the meta-training procedure have precluded application to real systems. In this work, we observe that the same latent space models which vastly improve the efficiency of end-to-end single-task RL can also be used to enable efficient meta-learning. By viewing the unknown task information as a hidden variable to be estimated from experience, we cast meta-RL into the framework of latent state estimation. Leveraging this insight, we present a practical and efficient method for the fast acquisition of robotic skills directly from raw sensory inputs. We show that our approach outperforms prior work on simulated continuous control tasks, and we apply our method to a real-world robotic manipulation task of peg insertion.