Timezone: »

 
Poster
EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine
Jiayi Weng · Min Lin · Shengyi Huang · Bo Liu · Denys Makoviichuk · Viktor Makoviychuk · Zichen Liu · Yufan Song · Ting Luo · Yukun Jiang · Zhongwen Xu · Shuicheng Yan

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #1026
in Poster Session 2 »

There has been significant progress in developing reinforcement learning (RL) training systems. Past works such as IMPALA, Apex, Seed RL, Sample Factory, and others, aim to improve the system's overall throughput. In this paper, we aim to address a common bottleneck in the RL training system, i.e., parallel environment execution, which is often the slowest part of the whole system but receives little attention. With a curated design for paralleling RL environments, we have improved the RL environment simulation speed across different hardware setups, ranging from a laptop and a modest workstation, to a high-end machine such as NVIDIA DGX-A100. On a high-end machine, EnvPool achieves one million frames per second for the environment execution on Atari environments and three million frames per second on MuJoCo environments. When running EnvPool on a laptop, the speed is 2.8x that of the Python subprocess. Moreover, great compatibility with existing RL training libraries has been demonstrated in the open-sourced community, including CleanRL, rl_games, DeepMind Acme, etc. Finally, EnvPool allows researchers to iterate their ideas at a much faster pace and has great potential to become the de facto RL environment execution engine. Example runs show that it only takes five minutes to train agents to play Atari Pong and MuJoCo Ant on a laptop. EnvPool is open-sourced at https://github.com/sail-sg/envpool.

Keywords: Reinforcement Learning · speed up

Author Information

Jiayi Weng (OpenAI)
Jiayi Weng

github.com/Trinkle23897 Creater of Tianshou and EnvPool, now working at OpenAI

Min Lin (Sea AI Lab)
Shengyi Huang (Drexel University)
Bo Liu (Peking University)
Denys Makoviichuk (Snap Inc)
Viktor Makoviychuk (NVIDIA)
Zichen Liu (national university of singaore, National University of Singapore)
Yufan Song (School of Computer Science, Carnegie Mellon University)
Ting Luo (CMU, Carnegie Mellon University)
Yukun Jiang (School of Computer Science, Carnegie Mellon University)
Zhongwen Xu (Sea AI Lab)
Shuicheng Yan (Sea AI Lab)

More from the Same Authors