Timezone: »

 
Poster
Risk-Aware Transfer in Reinforcement Learning using Successor Features
Michael Gimelfarb · Andre Barreto · Scott Sanner · Chi-Guhn Lee

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @ Virtual
in Poster Session 6 »

Sample efficiency and risk-awareness are central to the development of practical reinforcement learning (RL) for complex decision-making. The former can be addressed by transfer learning, while the latter by optimizing some utility function of the return. However, the problem of transferring skills in a risk-aware manner is not well-understood. In this paper, we address the problem of transferring policies between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams. Our approach begins by extending the idea of generalized policy improvement to maximize entropic utilities, thus extending the dynamic programming's policy improvement operation to sets of policies \emph{and} levels of risk-aversion. Next, we extend the idea of successor features (SF), a value function representation that decouples the environment dynamics from the rewards, to capture the variance of returns. Our resulting risk-aware successor features (RaSF) integrate seamlessly within the RL framework, inherit the superior task generalization ability of SFs, while incorporating risk into the decision-making. Experiments on a discrete navigation domain and control of a simulated robotic arm demonstrate the ability of RaSFs to outperform alternative methods including SFs, when taking the risk of the learned policies into account.

Keywords: Reinforcement Learning and Planning · Representation Learning · Transfer Learning

Author Information

Michael Gimelfarb (University of Toronto)
Andre Barreto (DeepMind)
Scott Sanner (University of Toronto)
Chi-Guhn Lee (University of Toronto)

More from the Same Authors