Timezone: »

 
Poster
Understanding End-to-End Model-Based Reinforcement Learning Methods as Implicit Parameterization
Clement Gehring · Kenji Kawaguchi · Jiaoyang Huang · Leslie Kaelbling

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @
in Poster Session 6 »

Estimating the per-state expected cumulative rewards is a critical aspect of reinforcement learning approaches, however the experience is obtained, but standard deep neural-network function-approximation methods are often inefficient in this setting. An alternative approach, exemplified by value iteration networks, is to learn transition and reward models of a latent Markov decision process whose value predictions fit the data. This approach has been shown empirically to converge faster to a more robust solution in many cases, but there has been little theoretical study of this phenomenon. In this paper, we explore such implicit representations of value functions via theory and focused experimentation. We prove that, for a linear parametrization, gradient descent converges to global optima despite non-linearity and non-convexity introduced by the implicit representation. Furthermore, we derive convergence rates for both cases which allow us to identify conditions under which stochastic gradient descent (SGD) with this implicit representation converges substantially faster than its explicit counterpart. Finally, we provide empirical results in some simple domains that illustrate the theoretical findings.

Keywords: Theory · Deep Learning · Optimization · Reinforcement Learning and Planning

Author Information

Clement Gehring (Massachusetts Institute of Technology)
Kenji Kawaguchi (MIT)
Jiaoyang Huang (New York University)
Leslie Kaelbling (MIT)

More from the Same Authors