Timezone: »

 
Poster
Enhancing Safe Exploration Using Safety State Augmentation
Aivar Sootla · Alexander Cowen-Rivers · Jun Wang · Haitham Bou Ammar

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #325

Safe exploration is a challenging and important problem in model-free reinforcement learning (RL). Often the safety cost is sparse and unknown, which unavoidably leads to constraint violations - a phenomenon ideally to be avoided in safety-critical applications. We tackle this problem by augmenting the state-space with a safety state, which is nonnegative if and only if the constraint is satisfied. The value of this state also serves as a distance toward constraint violation, while its initial value indicates the available safety budget. This idea allows us to derive policies for scheduling the safety budget during training. We call our approach Simmer (Safe policy IMproveMEnt for RL) to reflect the careful nature of these schedules. We apply this idea to two safe RL problems: RL with constraints imposed on an average cost, and RL with constraints imposed on a cost with probability one. Our experiments suggest that "simmering" a safe algorithm can improve safety during training for both settings. We further show that Simmer can stabilize training and improve the performance of safe RL with average constraints.

Author Information

Aivar Sootla (Byju's Lab)
Alexander Cowen-Rivers (University College London)
Jun Wang (UCL)
Haitham Bou Ammar (Huawei R&D UK)

More from the Same Authors

  • 2022 Poster: Multiagent Q-learning with Sub-Team Coordination »
    Wenhan Huang · Kai Li · Kun Shao · Tianze Zhou · Matthew Taylor · Jun Luo · Dongge Wang · Hangyu Mao · Jianye Hao · Jun Wang · Xiaotie Deng
  • 2022 Poster: M2N: Mesh Movement Networks for PDE Solvers »
    Wenbin Song · Mingrui Zhang · Joseph G Wallwork · Junpeng Gao · Zheng Tian · Fanglei Sun · Matthew Piggott · Junqing Chen · Zuoqiang Shi · Xiang Chen · Jun Wang
  • 2022 : Structured Q-learning For Antibody Design »
    Alexander Cowen-Rivers · Philip John Gorinski · aivar sootla · Asif Khan · Jun WANG · Jan Peters · Haitham Bou Ammar
  • 2022 : Structured Q-learning For Antibody Design »
    Alexander Cowen-Rivers · Philip John Gorinski · aivar sootla · Asif Khan · Jun WANG · Jan Peters · Haitham Bou Ammar
  • 2022 : Contextual Transformer for Offline Meta Reinforcement Learning »
    Runji Lin · Ye Li · Xidong Feng · Zhaowei Zhang · XIAN HONG WU FUNG · Haifeng Zhang · Jun Wang · Yali Du · Yaodong Yang
  • 2022 Spotlight: Lightning Talks 5A-3 »
    Minting Pan · Xiang Chen · Wenhan Huang · Can Chang · Zhecheng Yuan · Jianzhun Shao · Yushi Cao · Peihao Chen · Ke Xue · Zhengrong Xue · Zhiqiang Lou · Xiangming Zhu · Lei Li · Zhiming Li · Kai Li · Jiacheng Xu · Dongyu Ji · Ni Mu · Kun Shao · Tianpei Yang · Kunyang Lin · Ningyu Zhang · Yunbo Wang · Lei Yuan · Bo Yuan · Hongchang Zhang · Jiajun Wu · Tianze Zhou · Xueqian Wang · Ling Pan · Yuhang Jiang · Xiaokang Yang · Xiaozhuan Liang · Hao Zhang · Weiwen Hu · Miqing Li · YAN ZHENG · Matthew Taylor · Huazhe Xu · Shumin Deng · Chao Qian · YI WU · Shuncheng He · Wenbing Huang · Chuanqi Tan · Zongzhang Zhang · Yang Gao · Jun Luo · Yi Li · Xiangyang Ji · Thomas Li · Mingkui Tan · Fei Huang · Yang Yu · Huazhe Xu · Dongge Wang · Jianye Hao · Chuang Gan · Yang Liu · Luo Si · Hangyu Mao · Huajun Chen · Jianye Hao · Jun Wang · Xiaotie Deng
  • 2022 Spotlight: Multiagent Q-learning with Sub-Team Coordination »
    Wenhan Huang · Kai Li · Kun Shao · Tianze Zhou · Matthew Taylor · Jun Luo · Dongge Wang · Hangyu Mao · Jianye Hao · Jun Wang · Xiaotie Deng
  • 2022 Spotlight: Optimistic Tree Searches for Combinatorial Black-Box Optimization »
    Cedric Malherbe · Antoine Grosnit · Rasul Tutunov · Haitham Bou Ammar · Jun Wang
  • 2022 Poster: Optimistic Tree Searches for Combinatorial Black-Box Optimization »
    Cedric Malherbe · Antoine Grosnit · Rasul Tutunov · Haitham Bou Ammar · Jun Wang
  • 2022 Poster: Multi-Agent Reinforcement Learning is a Sequence Modeling Problem »
    Muning Wen · Jakub Kuba · Runji Lin · Weinan Zhang · Ying Wen · Jun Wang · Yaodong Yang
  • 2022 Poster: A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning »
    Bo Liu · Xidong Feng · Jie Ren · Luo Mai · Rui Zhu · Haifeng Zhang · Jun Wang · Yaodong Yang
  • 2019 : Extended Poster Session »
    Travis LaCroix · Marie Ossenkopf · Mina Lee · Nicole Fitzgerald · Daniela Mihai · Jonathon Hare · Ali Zaidi · Alexander Cowen-Rivers · Alana Marzoev · Eugene Kharitonov · Luyao Yuan · Tomasz Korbak · Paul Pu Liang · Yi Ren · Roberto Dessì · Peter Potash · Shangmin Guo · Tatsunori Hashimoto · Percy Liang · Julian Zubek · Zipeng Fu · Song-Chun Zhu · Adam Lerer
  • 2019 : Contributed Talk - 2 »
    Alexander Cowen-Rivers