Timezone: »

 
Poster
High-Throughput Synchronous Deep RL
Iou-Jen Liu · Raymond A. Yeh · Alex Schwing

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #504
in Poster Session 2 »

Various parallel actor-learner methods reduce long training times for deep reinforcement learning. Synchronous methods enjoy training stability while having lower data throughput. In contrast, asynchronous methods achieve high throughput but suffer from stability issues and lower sample efficiency due to ‘stale policies.’ To combine the advantages of both methods we propose High-Throughput Synchronous Deep Reinforcement Learning (HTS-RL). In HTS-RL, we perform learning and rollouts concurrently, devise a system design which avoids ‘stale policies’ and ensure that actors interact with environment replicas in an asynchronous manner while maintaining full determinism. We evaluate our approach on Atari games and the Google Research Football environment. Compared to synchronous baselines, HTS-RL is 2−6X faster. Compared to state-of-the-art asynchronous methods, HTS-RL has competitive throughput and consistently achieves higher average episode rewards.

Author Information

Iou-Jen Liu (University of Illinois at Urbana-Champaign)
Raymond A. Yeh (University of Illinois at Urbana–Champaign)
Alex Schwing (University of Illinois at Urbana-Champaign)

More from the Same Authors