Timezone: »

 
Poster
IQ-Learn: Inverse soft-Q Learning for Imitation
Divyansh Garg · Shuvam Chakraborty · Chris Cundy · Jiaming Song · Stefano Ermon

Wed Dec 08 12:30 AM -- 02:00 AM (PST) @ Virtual
in Poster Session 3 »

In many sequential decision-making problems (e.g., robotics control, game playing, sequential prediction), human or expert data is available containing useful information about the task. However, imitation learning (IL) from a small amount of expert data can be challenging in high-dimensional environments with complex dynamics. Behavioral cloning is a simple method that is widely used due to its simplicity of implementation and stable convergence but doesn't utilize any information involving the environment’s dynamics. Many existing methods that exploit dynamics information are difficult to train in practice due to an adversarial optimization process over reward and policy approximators or biased, high variance gradient estimators. We introduce a method for dynamics-aware IL which avoids adversarial training by learning a single Q-function, implicitly representing both reward and policy. On standard benchmarks, the implicitly learned rewards show a high positive correlation with the ground-truth rewards, illustrating our method can also be used for inverse reinforcement learning (IRL). Our method, Inverse soft-Q learning (IQ-Learn) obtains state-of-the-art results in offline and online imitation learning settings, significantly outperforming existing methods both in the number of required environment interactions and scalability in high-dimensional spaces, often by more than 3x.

Keywords: Optimization · Reinforcement Learning and Planning · Adversarial Robustness and Security

Author Information

Divyansh Garg (Stanford University)
Shuvam Chakraborty (Stanford University)
Chris Cundy (Stanford University)
Jiaming Song (Stanford University)

I am a first year Ph.D. student in Stanford University. I think about problems in machine learning and deep learning under the supervision of Stefano Ermon. I did my undergrad at Tsinghua University, where I was lucky enough to collaborate with Jun Zhu and Lawrence Carin on scalable Bayesian machine learning.

Stefano Ermon (Stanford)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors