Timezone: »

 
Spotlight
Learning Compositional Neural Programs with Recursive Tree Search and Planning
Thomas PIERROT · Guillaume Ligner · Scott Reed · Olivier Sigaud · Nicolas Perrin · Alexandre Laterre · David Kas · Karim Beguir · Nando de Freitas

Tue Dec 10 04:40 PM -- 04:45 PM (PST) @ West Ballroom A + B
in Track 3 Session 2 »

We propose a novel reinforcement learning algorithm, AlphaNPI, that incorpo- rates the strengths of Neural Programmer-Interpreters (NPI) and AlphaZero. NPI contributes structural biases in the form of modularity, hierarchy and recursion, which are helpful to reduce sample complexity, improve generalization and in- crease interpretability. AlphaZero contributes powerful neural network guided search algorithms, which we augment with recursion. AlphaNPI only assumes a hierarchical program specification with sparse rewards: 1 when the program execution satisfies the specification, and 0 otherwise. This specification enables us to overcome the need for strong supervision in the form of execution traces and consequently train NPI models effectively with reinforcement learning. The experiments show that AlphaNPI can sort as well as previous strongly supervised NPI variants. The AlphaNPI agent is also trained on a Tower of Hanoi puzzle with two disks and is shown to generalize to puzzles with an arbitrary number of disks. The experiments also show that when deploying our neural network policies, it is advantageous to do planning with guided Monte Carlo tree search.

[ Paper]  [ Poster]  [ Slides] 

Author Information

Thomas PIERROT (InstaDeep)

PhD candidate at InstaDeep and Paris Sorbonne University.

Guillaume Ligner (InstaDeep)
Scott Reed (Google DeepMind)
Olivier Sigaud (Sorbonne University)
Nicolas Perrin (CNRS, Sorbonne Université)
Alexandre Laterre (InstaDeep)
David Kas (Entrepreneur First)

Tech Entrepreneur / AI Research and Engineering

Karim Beguir (InstaDeep)
Nando de Freitas (DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors