Timezone: »

 
Poster
Robust Imitation via Mirror Descent Inverse Reinforcement Learning
Dong-Sig Han · Hyunseo Kim · Hyundo Lee · JeHwan Ryu · Byoung-Tak Zhang

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #807
in Poster Session 4 »
Recently, adversarial imitation learning has shown a scalable reward acquisition method for inverse reinforcement learning (IRL) problems. However, estimated reward signals often become uncertain and fail to train a reliable statistical model since the existing methods tend to solve hard optimization problems directly. Inspired by a first-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions for a constrained convex problem. IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount of reward learning is regulated with respect to local geometric constraints. We prove that the proposed mirror descent update rule ensures robust minimization of a Bregman divergence in terms of a rigorous regret bound of $\mathcal{O}(1/T)$ for step sizes $\{\eta_t\}_{t=1}^{T}$. Our IRL method was applied on top of an adversarial framework, and it outperformed existing adversarial methods in an extensive suite of benchmarks.
Keywords: Learning Theory · imitation learning · inverse reinforcement learning · regularized Markov decision processes
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Dong-Sig Han (Seoul National University)
Hyunseo Kim (Seoul National University)
Hyundo Lee (Seoul National University)
JeHwan Ryu (Seoul National University)
Byoung-Tak Zhang (Seoul National University)

More from the Same Authors