Timezone: »

 
Poster
A Geometric Perspective on Optimal Representations for Reinforcement Learning
Marc Bellemare · Will Dabney · Robert Dadashi · Adrien Ali Taiga · Pablo Samuel Castro · Nicolas Le Roux · Dale Schuurmans · Tor Lattimore · Clare Lyle

Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #200
in Reinforcement Learning and Planning -- Reinforcement Learning »

We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. From there, we provide formal evidence regarding the usefulness of value functions as auxiliary tasks in reinforcement learning. Our formulation considers adapting the representation to minimize the (linear) approximation of the value function of all stationary policies for a given environment. We show that this optimization reduces to making accurate predictions regarding a special class of value functions which we call adversarial value functions (AVFs). We demonstrate that using value functions as auxiliary tasks corresponds to an expected-error relaxation of our formulation, with AVFs a natural candidate, and identify a close relationship with proto-value functions (Mahadevan, 2005). We highlight characteristics of AVFs and their usefulness as auxiliary tasks in a series of experiments on the four-room domain.

Keywords: Reinforcement Learning and Planning · Markov Decision Processes · Reinforcement Learning and Planning
[ Paper]  [ Slides] 

Author Information

Marc Bellemare (Google Brain)
Will Dabney (DeepMind)
Robert Dadashi (Google Brain)
Adrien Ali Taiga (MILA)
Pablo Samuel Castro (Google)
Nicolas Le Roux (Google Brain)
Dale Schuurmans (Google Inc.)
Tor Lattimore (DeepMind)
Clare Lyle (University of Oxford)

More from the Same Authors