Timezone: »

 
Poster
Program Synthesis Guided Reinforcement Learning for Partially Observed Environments
Yichen Yang · Jeevana Priya Inala · Osbert Bastani · Yewen Pu · Armando Solar-Lezama · Martin Rinard

Wed Dec 08 04:30 PM -- 06:00 PM (PST) @
in Poster Session 4 »

A key challenge for reinforcement learning is solving long-horizon planning problems. Recent work has leveraged programs to guide reinforcement learning in these settings. However, these approaches impose a high manual burden on the user since they must provide a guiding program for every new task. Partially observed environments further complicate the programming task because the program must implement a strategy that correctly, and ideally optimally, handles every possible configuration of the hidden regions of the environment. We propose a new approach, model predictive program synthesis (MPPS), that uses program synthesis to automatically generate the guiding programs. It trains a generative model to predict the unobserved portions of the world, and then synthesizes a program based on samples from this model in a way that is robust to its uncertainty. In our experiments, we show that our approach significantly outperforms non-program-guided approaches on a set of challenging benchmarks, including a 2D Minecraft-inspired environment where the agent must complete a complex sequence of subtasks to achieve its goal, and achieves a similar performance as using handcrafted programs to guide the agent. Our results demonstrate that our approach can obtain the benefits of program-guided reinforcement learning without requiring the user to provide a new guiding program for every new task.

Keywords: Reinforcement Learning and Planning · Generative Model

Author Information

Yichen Yang (MIT)
Jeevana Priya Inala (Microsoft Research)
Osbert Bastani (University of Pennsylvania)
Yewen Pu (Autodesk)
Armando Solar-Lezama (MIT)
Martin Rinard (MIT)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors