Timezone: »
We present temporally abstract actor-critic (TAAC), a simple but effective off-policy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framework. TAAC adds a second-stage binary policy to choose between the previous action and a new action output by an actor. Crucially, its "act-or-repeat" decision hinges on the actually sampled action instead of the expected behavior of the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features: a) persistent exploration, and b) a new compare-through Q operator for multi-step TD backup, specially tailored to the action repetition scenario. We demonstrate TAAC's advantages over several strong baselines across 14 continuous control tasks. Our surprising finding reveals that while achieving top performance, TAAC is able to "mine" a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition. This suggests that aside from encouraging persistent exploration, action repetition can find its place in a good policy behavior. Code is available at https://github.com/hnyu/taac.
Author Information
Haonan Yu (Horizon Robotics)
Wei Xu (Horizon Robotics)
Haichao Zhang (Horizon Robotics)
More from the Same Authors
-
2022 Poster: Towards Safe Reinforcement Learning with a Safety Editor Policy »
Haonan Yu · Wei Xu · Haichao Zhang -
2022 Poster: Society of Agents: Regret Bounds of Concurrent Thompson Sampling »
Yan Chen · Perry Dong · Qinxun Bai · Maria Dimakopoulou · Wei Xu · Zhengyuan Zhou -
2022 Poster: PaCo: Parameter-Compositional Multi-task Reinforcement Learning »
Lingfeng Sun · Haichao Zhang · Wei Xu · Masayoshi TOMIZUKA -
2019 Poster: Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training »
Haichao Zhang · Jianyu Wang -
2018 Poster: Adversarial Text Generation via Feature-Mover's Distance »
Liqun Chen · Shuyang Dai · Chenyang Tao · Haichao Zhang · Zhe Gan · Dinghan Shen · Yizhe Zhang · Guoyin Wang · Dinghan Shen · Lawrence Carin -
2017 : Break + Poster (1) »
Devendra Singh Chaplot · CHIH-YAO MA · Simon Brodeur · Eri Matsuo · Ichiro Kobayashi · Seitaro Shinagawa · Koichiro Yoshino · Yuhong Guo · Ben Murdoch · Kanthashree Mysore Sathyendra · Daniel Ricks · Haichao Zhang · Joshua Peterson · Li Zhang · Mircea Mironenco · Peter Anderson · Mark Johnson · Kang Min Yoo · Guntis Barzdins · Ahmed H Zaidi · Martin Andrews · Sam Witteveen · SUBBAREDDY OOTA · Prashanth Vijayaraghavan · Ke Wang · Yan Zhu · Renars Liepins · Max Quinn · Amit Raj · Vincent Cartillier · Eric Chu · Ethan Caballero · Fritz Obermeyer -
2014 Poster: Scale Adaptive Blind Deblurring »
Haichao Zhang · Jianchao Yang -
2013 Poster: Non-Uniform Camera Shake Removal Using a Spatially-Adaptive Sparse Penalty »
Haichao Zhang · David Wipf -
2013 Oral: Non-Uniform Camera Shake Removal Using a Spatially-Adaptive Sparse Penalty »
Haichao Zhang · David Wipf