Timezone: »

 
Oral
Actor-Critic Algorithms for Risk-Sensitive MDPs
Prashanth L.A. · Mohammad Ghavamzadeh

Sun Dec 08 09:50 AM -- 10:10 AM (PST) @ Harvey's Convention Center Floor, CC
in Oral Session 9 »

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms for estimating the gradient and updating the policy parameters in the ascent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.

Author Information

Prashanth L.A. (INRIA)
Mohammad Ghavamzadeh (Facebook AI Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors