Timezone: »

 
Poster
Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates
Carlos Riquelme · Hugo Penedones · Damien Vincent · Hartmut Maennel · Sylvain Gelly · Timothy A Mann · Andre Barreto · Gergely Neu

Thu Dec 12 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #224
in Reinforcement Learning and Planning -- Reinforcement Learning »

We consider the core reinforcement-learning problem of on-policy value function approximation from a batch of trajectory data, and focus on various issues of Temporal Difference (TD) learning and Monte Carlo (MC) policy evaluation. The two methods are known to achieve complementary bias-variance trade-off properties, with TD tending to achieve lower variance but potentially higher bias. In this paper, we argue that the larger bias of TD can be a result of the amplification of local approximation errors. We address this by proposing an algorithm that adaptively switches between TD and MC in each state, thus mitigating the propagation of errors. Our method is based on learned confidence intervals that detect biases of TD estimates. We demonstrate in a variety of policy evaluation tasks that this simple adaptive algorithm performs competitively with the best approach in hindsight, suggesting that learned confidence intervals are a powerful technique for adapting policy evaluation to use TD or MC returns in a data-driven way.

Keywords: Reinforcement Learning · Markov Decision Processes · Reinforcement Learning and Planning · Algorithms -> Uncertainty Estimation; Reinforcement Learning and Planning
[ Paper]  [ Poster] 

Author Information

Carlos Riquelme (Google Brain)
Hugo Penedones (Google DeepMind)
Damien Vincent (Google Brain)
Hartmut Maennel (Google)
Sylvain Gelly (Google Brain (Zurich))
Timothy A Mann (DeepMind)
Andre Barreto (DeepMind)
Gergely Neu (Universitat Pompeu Fabra)

More from the Same Authors