Timezone: »

 
Poster
Reinforcement Learning with Non-Exponential Discounting
Matthias Schultheis · Constantin Rothkopf · Heinz Koeppl

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #507
in Poster Session 4 »

Commonly in reinforcement learning (RL), rewards are discounted over time using an exponential function to model time preference, thereby bounding the expected long-term reward. In contrast, in economics and psychology, it has been shown that humans often adopt a hyperbolic discounting scheme, which is optimal when a specific task termination time distribution is assumed. In this work, we propose a theory for continuous-time model-based reinforcement learning generalized to arbitrary discount functions. This formulation covers the case in which there is a non-exponential random termination time. We derive a Hamilton–Jacobi–Bellman (HJB) equation characterizing the optimal policy and describe how it can be solved using a collocation method, which uses deep learning for function approximation. Further, we show how the inverse RL problem can be approached, in which one tries to recover properties of the discount function given decision data. We validate the applicability of our proposed approach on two simulated problems. Our approach opens the way for the analysis of human discounting in sequential decision-making tasks.

Keywords: Reinforcement Learning · continuous time · discounting · human decision making · inverse reinforcement learning · optimal control

Author Information

Matthias Schultheis (Technische Universität Darmstadt)
Constantin Rothkopf (TU Darmstadt)
Heinz Koeppl (Technische Universität Darmstadt)

More from the Same Authors