Timezone: »
Various parallel actor-learner methods reduce long training times for deep reinforcement learning. Synchronous methods enjoy training stability while having lower data throughput. In contrast, asynchronous methods achieve high throughput but suffer from stability issues and lower sample efficiency due to ‘stale policies.’ To combine the advantages of both methods we propose High-Throughput Synchronous Deep Reinforcement Learning (HTS-RL). In HTS-RL, we perform learning and rollouts concurrently, devise a system design which avoids ‘stale policies’ and ensure that actors interact with environment replicas in an asynchronous manner while maintaining full determinism. We evaluate our approach on Atari games and the Google Research Football environment. Compared to synchronous baselines, HTS-RL is 2−6X faster. Compared to state-of-the-art asynchronous methods, HTS-RL has competitive throughput and consistently achieves higher average episode rewards.
Author Information
Iou-Jen Liu (University of Illinois at Urbana-Champaign)
Raymond A. Yeh (University of Illinois at Urbana–Champaign)
Alex Schwing (University of Illinois at Urbana-Champaign)
More from the Same Authors
-
2021 Spotlight: Per-Pixel Classification is Not All You Need for Semantic Segmentation »
Bowen Cheng · Alex Schwing · Alexander Kirillov -
2022 Poster: CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations »
Kai Yan · Alex Schwing · Yu-Xiong Wang -
2022 Poster: DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data »
Tiantian Fang · Ruoyu Sun · Alex Schwing -
2022 Poster: Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks »
Renan A. Rojas-Gomez · Teck-Yian Lim · Alex Schwing · Minh Do · Raymond A. Yeh -
2022 Poster: On the Importance of Gradient Norm in PAC-Bayesian Bounds »
Itai Gat · Yossi Adi · Alex Schwing · Tamir Hazan -
2021 Poster: Bridging the Imitation Gap by Adaptive Insubordination »
Luca Weihs · Unnat Jain · Iou-Jen Liu · Jordi Salvador · Svetlana Lazebnik · Aniruddha Kembhavi · Alex Schwing -
2021 Poster: Per-Pixel Classification is Not All You Need for Semantic Segmentation »
Bowen Cheng · Alex Schwing · Alexander Kirillov -
2021 Poster: A Contrastive Learning Approach for Training Variational Autoencoder Priors »
Jyoti Aneja · Alex Schwing · Jan Kautz · Arash Vahdat -
2021 Poster: Class-agnostic Reconstruction of Dynamic Objects from Videos »
Zhongzheng Ren · Xiaoming Zhao · Alex Schwing -
2021 Poster: Perceptual Score: What Data Modalities Does Your Model Perceive? »
Itai Gat · Idan Schwartz · Alex Schwing -
2020 Poster: Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning »
Zhongzheng Ren · Raymond A. Yeh · Alex Schwing -
2020 Poster: Towards a Better Global Loss Landscape of GANs »
Ruoyu Sun · Tiantian Fang · Alex Schwing -
2020 Oral: Towards a Better Global Loss Landscape of GANs »
Ruoyu Sun · Tiantian Fang · Alex Schwing -
2020 Session: Orals & Spotlights Track 22: Vision Applications »
Leonid Sigal · Alex Schwing -
2020 Poster: Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies »
Itai Gat · Idan Schwartz · Alex Schwing · Tamir Hazan -
2019 : Poster Session »
Pravish Sainath · Mohamed Akrout · Charles Delahunt · Nathan Kutz · Guangyu Robert Yang · Joseph Marino · L F Abbott · Nicolas Vecoven · Damien Ernst · andrew warrington · Michael Kagan · Kyunghyun Cho · Kameron Harris · Leopold Grinberg · John J. Hopfield · Dmitry Krotov · Taliah Muhammad · Erick Cobos · Edgar Walker · Jacob Reimer · Andreas Tolias · Alexander Ecker · Janaki Sheth · Yu Zhang · Maciej Wołczyk · Jacek Tabor · Szymon Maszke · Roman Pogodin · Dane Corneil · Wulfram Gerstner · Baihan Lin · Guillermo Cecchi · Jenna M Reinen · Irina Rish · Guillaume Bellec · Darjan Salaj · Anand Subramoney · Wolfgang Maass · Yueqi Wang · Ari Pakman · Jin Hyung Lee · Liam Paninski · Bryan Tripp · Colin Graber · Alex Schwing · Luke Prince · Gabriel Ocker · Michael Buice · Benjamin Lansdell · Konrad Kording · Jack Lindsey · Terrence Sejnowski · Matthew Farrell · Eric Shea-Brown · Nicolas Farrugia · Victor Nepveu · Jiwoong Im · Kristin Branson · Brian Hu · Ramakrishnan Iyer · Stefan Mihalas · Sneha Aenugu · Hananel Hazan · Sihui Dai · Tan Nguyen · Doris Tsao · Richard Baraniuk · Anima Anandkumar · Hidenori Tanaka · Aran Nayebi · Stephen Baccus · Surya Ganguli · Dean Pospisil · Eilif Muller · Jeffrey S Cheng · Gaël Varoquaux · Kamalaker Dadi · Dimitrios C Gklezakos · Rajesh PN Rao · Anand Louis · Christos Papadimitriou · Santosh Vempala · Naganand Yadati · Daniel Zdeblick · Daniela M Witten · Nicholas Roberts · Vinay Prabhu · Pierre Bellec · Poornima Ramesh · Jakob H Macke · Santiago Cadena · Guillaume Bellec · Franz Scherr · Owen Marschall · Robert Kim · Hannes Rapp · Marcio Fonseca · Oliver Armitage · Jiwoong Im · Thomas Hardcastle · Abhishek Sharma · Wyeth Bair · Adrian Valente · Shane Shang · Merav Stern · Rutuja Patil · Peter Wang · Sruthi Gorantla · Peter Stratton · Tristan Edwards · Jialin Lu · Martin Ester · Yurii Vlasov · Siavash Golkar -
2019 : Coffee Break & Poster Session 2 »
Juho Lee · Yoonho Lee · Yee Whye Teh · Raymond A. Yeh · Yuan-Ting Hu · Alex Schwing · Sara Ahmadian · Alessandro Epasto · Marina Knittel · Ravi Kumar · Mohammad Mahdian · Christian Bueno · Aditya Sanghi · Pradeep Kumar Jayaraman · Ignacio Arroyo-Fernández · Andrew Hryniowski · Vinayak Mathur · Sanjay Singh · Shahrzad Haddadan · Vasco Portilheiro · Luna Zhang · Mert Yuksekgonul · Jhosimar Arias Figueroa · Deepak Maurya · Balaraman Ravindran · Frank NIELSEN · Philip Pham · Justin Payan · Andrew McCallum · Jinesh Mehta · Ke SUN -
2019 : Contributed Talk - Chirality Nets: Exploiting Structure in Human Pose Regression »
Raymond A. Yeh · Yuan-Ting Hu · Alex Schwing -
2019 : Poster Session »
Jonathan Scarlett · Piotr Indyk · Ali Vakilian · Adrian Weller · Partha P Mitra · Benjamin Aubin · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová · Kristina Monakhova · Joshua Yurtsever · Laura Waller · Hendrik Sommerhoff · Michael Moeller · Rushil Anirudh · Shuang Qiu · Xiaohan Wei · Zhuoran Yang · Jayaraman Thiagarajan · Salman Asif · Michael Gillhofer · Johannes Brandstetter · Sepp Hochreiter · Felix Petersen · Dhruv Patel · Assad Oberai · Akshay Kamath · Sushrut Karmalkar · Eric Price · Ali Ahmed · Zahra Kadkhodaie · Sreyas Mohan · Eero Simoncelli · Carlos Fernandez-Granda · Oscar Leong · Wesam Sakla · Rebecca Willett · Stephan Hoyer · Jascha Sohl-Dickstein · Sam Greydanus · Gauri Jagatap · Chinmay Hegde · Michael Kellman · Jonathan Tamir · Nouamane Laanait · Ousmane Dia · Mirco Ravanelli · Jonathan Binas · Negar Rostamzadeh · Shirin Jalali · Tiantian Fang · Alex Schwing · Sébastien Lachapelle · Philippe Brouillard · Tristan Deleu · Simon Lacoste-Julien · Stella Yu · Arya Mazumdar · Ankit Singh Rawat · Yue Zhao · Jianshu Chen · Xiaoyang Li · Hubert Ramsauer · Gabrio Rizzuti · Nikolaos Mitsakos · Dingzhou Cao · Thomas Strohmer · Yang Li · Pei Peng · Gregory Ongie -
2019 Poster: Chirality Nets for Human Pose Regression »
Raymond A. Yeh · Yuan-Ting Hu · Alex Schwing -
2019 Poster: Graph Structured Prediction Energy Networks »
Colin Graber · Alex Schwing -
2019 Poster: TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines »
Jingxiang Lin · Unnat Jain · Alex Schwing -
2019 Poster: Co-Generation with GANs using AIS based HMC »
Tiantian Fang · Alex Schwing -
2018 Poster: Deep Structured Prediction with Nonlinear Output Transformations »
Colin Graber · Ofer Meshi · Alex Schwing -
2018 Poster: Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training »
Youjie Li · Mingchao Yu · Songze Li · Salman Avestimehr · Nam Sung Kim · Alex Schwing -
2018 Poster: Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering »
Medhini Narasimhan · Svetlana Lazebnik · Alex Schwing -
2018 Poster: GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training »
Mingchao Yu · Zhifeng Lin · Krishna Narra · Songze Li · Youjie Li · Nam Sung Kim · Alex Schwing · Murali Annavaram · Salman Avestimehr -
2017 Poster: Dualing GANs »
Yujia Li · Alex Schwing · Kuan-Chieh Wang · Richard Zemel -
2017 Spotlight: Dualing GANs »
Yujia Li · Alex Schwing · Kuan-Chieh Wang · Richard Zemel -
2017 Poster: MaskRNN: Instance Level Video Object Segmentation »
Yuan-Ting Hu · Jia-Bin Huang · Alex Schwing -
2017 Poster: Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts »
Raymond A. Yeh · Jinjun Xiong · Wen-Mei Hwu · Minh Do · Alex Schwing -
2017 Poster: Asynchronous Parallel Coordinate Minimization for MAP Inference »
Ofer Meshi · Alex Schwing -
2017 Oral: Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts »
Raymond A. Yeh · Jinjun Xiong · Wen-Mei Hwu · Minh Do · Alex Schwing -
2017 Poster: High-Order Attention Models for Visual Question Answering »
Idan Schwartz · Alex Schwing · Tamir Hazan -
2017 Poster: Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space »
Liwei Wang · Alex Schwing · Svetlana Lazebnik -
2016 Poster: Constraints Based Convex Belief Propagation »
Yaniv Tenzer · Alex Schwing · Kevin Gimpel · Tamir Hazan -
2016 Poster: Learning Deep Parsimonious Representations »
Renjie Liao · Alex Schwing · Richard Zemel · Raquel Urtasun -
2015 Poster: Smooth and Strong: MAP Inference with Linear Convergence »
Ofer Meshi · Mehrdad Mahdavi · Alex Schwing -
2014 Poster: Efficient Inference of Continuous Markov Random Fields with Polynomial Potentials »
Shenlong Wang · Alex Schwing · Raquel Urtasun -
2014 Poster: Message Passing Inference for Large Scale Graphical Models with High Order Potentials »
Jian Zhang · Alex Schwing · Raquel Urtasun -
2013 Poster: Latent Structured Active Learning »
Wenjie Luo · Alex Schwing · Raquel Urtasun -
2012 Poster: Globally Convergent Dual MAP LP Relaxation Solvers using Fenchel-Young Margins »
Alex Schwing · Tamir Hazan · Marc Pollefeys · Raquel Urtasun