Timezone: »

 
Poster
Conservative Offline Distributional Reinforcement Learning
Jason Yecheng Ma · Dinesh Jayaraman · Osbert Bastani

Thu Dec 09 04:30 PM -- 06:00 PM (PST) @
in Poster Session 7 »

Many reinforcement learning (RL) problems in practice are offline, learning purely from observational data. A key challenge is how to ensure the learned policy is safe, which requires quantifying the risk associated with different actions. In the online setting, distributional RL algorithms do so by learning the distribution over returns (i.e., cumulative rewards) instead of the expected return; beyond quantifying risk, they have also been shown to learn better representations for planning. We proposeConservative Offline Distributional Actor Critic (CODAC), an offline RL algorithm suitable for both risk-neutral and risk-averse domains. CODAC adapts distributional RL to the offline setting by penalizing the predicted quantiles of the return for out-of-distribution actions. We prove that CODAC learns a conservative return distribution---in particular, for finite MDPs, CODAC converges to an uniform lower bound on the quantiles of the return distribution; our proof relies on a novel analysis of the distributional Bellman operator. In our experiments, on two challenging robot navigation tasks, CODAC successfully learns risk-averse policies using offline data collected purely from risk-neutral agents. Furthermore, CODAC is state-of-the-art on the D4RL MuJoCo benchmark in terms of both expected and risk-sensitive performance.

Keywords: Reinforcement Learning and Planning

Author Information

Jason Yecheng Ma (University of Pennsylvania)
Dinesh Jayaraman (University of Pennsylvania)
Osbert Bastani (University of Pennsylvania)

More from the Same Authors