Timezone: »
Offline Reinforcement Learning (ORL) provides a framework to train control policies from fixed sub-optimal datasets, making it suitable for safety-critical applications like robotics. Despite significant algorithmic advances and benchmarking in simulation, the evaluation of ORL algorithms on real-world robot learning tasks has been limited. Since real robots are sensitive to details like sensor noises, reset conditions, demonstration sources, and test time distribution, it remains a question whether ORL is a competitive solution to real robotic challenges and what would characterize such tasks. We aim to address this deficiency through an empirical study of representative ORL algorithms on four table-top manipulation tasks using a Franka-Panda robot arm. Our evaluation finds that for scenarios with sufficient in-domain data of high quality, specialized ORL algorithms can be competitive with the behavior cloning approach. However, for scenarios that require out-of-distribution generalization or task transfer, ORL algorithms can learn and generalize from offline heterogeneous datasets and outperform behavior cloning. Project URL: https://sites.google.com/view/real-orl-anon
Author Information
Gaoyue Zhou (Carnegie Mellon University)
Liyiming Ke (Paul G Allen School of Computer Science & Engineering, University of Washington)
Siddhartha Srinivasa (Carnegie Mellon University)
Abhinav Gupta (Facebook AI Research/CMU)
Aravind Rajeswaran (FAIR)
Vikash Kumar (FAIR, Meta-AI)

I am currently a research scientist at Facebook AI Research (FAIR). I have also spent some time at Google-Brain, OpenAI and Berkeley Artificial Intelligence Research (BAIR) Lab. I did my PhD at CSE, University of Washington's Movement Control Lab, under the supervision of Prof. Emanuel Todorov and Prof. Sergey Levine. I am interested in the areas of Robotics, and Embodied Artificial Intelligence. My general interest lies in developing artificial agents that are cheap, portable and exhibit complex behaviors.
More from the Same Authors
-
2021 : RB2: Robotic Manipulation Benchmarking with a Twist »
Sudeep Dasari · Jianren Wang · Joyce Hong · Shikhar Bahl · Yixin Lin · Austin Wang · Abitha Thankaraj · Karanbir Chahal · Berk Calli · Saurabh Gupta · David Held · Lerrel Pinto · Deepak Pathak · Vikash Kumar · Abhinav Gupta -
2021 : KitchenShift: Evaluating Zero-Shot Generalization of Imitation-Based Policy Learning Under Domain Shifts »
Eliot Xing · Abhinav Gupta · Samantha Powers · Victoria Dean -
2021 : CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery »
Misha Laskin · Hao Liu · Xue Bin Peng · Denis Yarats · Aravind Rajeswaran · Pieter Abbeel -
2021 : Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL »
Catherine Cang · Aravind Rajeswaran · Pieter Abbeel · Misha Laskin -
2022 : Shared Hardware, Shared Baselines: An Offline Robotics Benchmark »
Gaoyue Zhou · Victoria Dean -
2022 : Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training »
Jason Yecheng Ma · Shagun Sodhani · Dinesh Jayaraman · Osbert Bastani · Vikash Kumar · Amy Zhang -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 : Real World Offline Reinforcement Learning with Realistic Data Source »
Gaoyue Zhou · Liyiming Ke · Siddhartha Srinivasa · Abhinav Gupta · Aravind Rajeswaran · Vikash Kumar -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 : Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training »
Jason Yecheng Ma · Shagun Sodhani · Dinesh Jayaraman · Osbert Bastani · Vikash Kumar · Amy Zhang -
2022 : Fifteen-minute Competition Overview Video »
Guillaume Durandau · Yuval Tassa · Vittorio Caggiano · Vikash Kumar · Seungmoon Song · Massimo Sartori · -
2022 : Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Grasps »
Sudeep Dasari · Vikash Kumar -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 : Offline Reinforcement Learning on Real Robot with Realistic Data Sources »
Gaoyue Zhou · Liyiming Ke · Siddhartha Srinivasa · Abhinav Gupta · Aravind Rajeswaran · Vikash Kumar -
2022 : Policy Architectures for Compositional Generalization in Control »
Allan Zhou · Vikash Kumar · Chelsea Finn · Aravind Rajeswaran -
2022 : MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations »
Nicklas Hansen · Yixin Lin · Hao Su · Xiaolong Wang · Vikash Kumar · Aravind Rajeswaran -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 : Real World Offline Reinforcement Learning with Realistic Data Source »
Gaoyue Zhou · Liyiming Ke · Siddhartha Srinivasa · Abhinav Gupta · Aravind Rajeswaran · Vikash Kumar -
2023 Competition: MyoChallenge 2023: Towards Human-Level Dexterity and Agility »
Vittorio Caggiano · · Guillaume Durandau · Seungmoon Song · Cameron Berg · Pierre Schumacher · Chun Kwang Tan · Massimo Sartori · Vikash Kumar -
2022 : Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training »
Jason Yecheng Ma · Shagun Sodhani · Dinesh Jayaraman · Osbert Bastani · Vikash Kumar · Amy Zhang -
2022 Competition: MyoChallenge: Learning contact-rich manipulation using a musculoskeletal hand »
Vittorio Caggiano · · Guillaume Durandau · Seungmoon Song · Yuval Tassa · Massimo Sartori · Vikash Kumar -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 Workshop: 3rd Offline Reinforcement Learning Workshop: Offline RL as a "Launchpad" »
Aviral Kumar · Rishabh Agarwal · Aravind Rajeswaran · Wenxuan Zhou · George Tucker · Doina Precup · Aviral Kumar -
2022 Poster: Unsupervised Reinforcement Learning with Contrastive Intrinsic Control »
Michael Laskin · Hao Liu · Xue Bin Peng · Denis Yarats · Aravind Rajeswaran · Pieter Abbeel -
2022 Poster: Learning State-Aware Visual Representations from Audible Interactions »
Himangi Mittal · Pedro Morgado · Unnat Jain · Abhinav Gupta -
2021 Oral: Interesting Object, Curious Agent: Learning Task-Agnostic Exploration »
Simone Parisi · Victoria Dean · Deepak Pathak · Abhinav Gupta -
2021 Poster: Visual Adversarial Imitation Learning using Variational Models »
Rafael Rafailov · Tianhe Yu · Aravind Rajeswaran · Chelsea Finn -
2021 Poster: No RL, No Simulation: Learning to Navigate without Navigating »
Meera Hahn · Devendra Singh Chaplot · Shubham Tulsiani · Mustafa Mukadam · James Rehg · Abhinav Gupta -
2021 Poster: COMBO: Conservative Offline Model-Based Policy Optimization »
Tianhe Yu · Aviral Kumar · Rafael Rafailov · Aravind Rajeswaran · Sergey Levine · Chelsea Finn -
2021 Poster: Interesting Object, Curious Agent: Learning Task-Agnostic Exploration »
Simone Parisi · Victoria Dean · Deepak Pathak · Abhinav Gupta -
2021 Poster: Decision Transformer: Reinforcement Learning via Sequence Modeling »
Lili Chen · Kevin Lu · Aravind Rajeswaran · Kimin Lee · Aditya Grover · Misha Laskin · Pieter Abbeel · Aravind Srinivas · Igor Mordatch -
2021 Poster: Reinforcement Learning with Latent Flow »
Wenling Shang · Xiaofei Wang · Aravind Srinivas · Aravind Rajeswaran · Yang Gao · Pieter Abbeel · Misha Laskin -
2020 : QA: Abhinav Gupta »
Abhinav Gupta -
2020 : Invited Talk: Abhinav Gupta »
Abhinav Gupta -
2020 Workshop: Object Representations for Learning and Reasoning »
William Agnew · Rim Assouel · Michael Chang · Antonia Creswell · Eliza Kosoy · Aravind Rajeswaran · Sjoerd van Steenkiste -
2020 Poster: Neural Dynamic Policies for End-to-End Sensorimotor Learning »
Shikhar Bahl · Mustafa Mukadam · Abhinav Gupta · Deepak Pathak -
2020 Poster: Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases »
Senthil Purushwalkam · Abhinav Gupta -
2020 Spotlight: Neural Dynamic Policies for End-to-End Sensorimotor Learning »
Shikhar Bahl · Mustafa Mukadam · Abhinav Gupta · Deepak Pathak -
2020 Poster: See, Hear, Explore: Curiosity via Audio-Visual Association »
Victoria Dean · Shubham Tulsiani · Abhinav Gupta -
2020 Poster: Object Goal Navigation using Goal-Oriented Semantic Exploration »
Devendra Singh Chaplot · Dhiraj Prakashchand Gandhi · Abhinav Gupta · Russ Salakhutdinov -
2020 Poster: MOReL: Model-Based Offline Reinforcement Learning »
Rahul Kidambi · Aravind Rajeswaran · Praneeth Netrapalli · Thorsten Joachims -
2019 Poster: Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller »
Pratyusha Sharma · Deepak Pathak · Abhinav Gupta -
2019 Poster: Meta-Learning with Implicit Gradients »
Aravind Rajeswaran · Chelsea Finn · Sham Kakade · Sergey Levine -
2018 Poster: Hardware Conditioned Policies for Multi-Robot Transfer Learning »
Tao Chen · Adithyavairavan Murali · Abhinav Gupta -
2018 Poster: Beyond Grids: Learning Graph Representations for Visual Recognition »
Yin Li · Abhinav Gupta -
2018 Poster: Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias »
Abhinav Gupta · Adithyavairavan Murali · Dhiraj Prakashchand Gandhi · Lerrel Pinto -
2017 Poster: Near-Optimal Edge Evaluation in Explicit Generalized Binomial Graphs »
Sanjiban Choudhury · Shervin Javdani · Siddhartha Srinivasa · Sebastian Scherer -
2017 Poster: Towards Generalization and Simplicity in Continuous Control »
Aravind Rajeswaran · Kendall Lowrey · Emanuel Todorov · Sham Kakade -
2016 : Invited Talk - Self Supervised Learning of Visual Representations »
Abhinav Gupta -
2016 : Abhinav Gupta »
Abhinav Gupta -
2016 : Abhinav Gupta »
Abhinav Gupta -
2013 Poster: Mid-level Visual Element Discovery as Discriminative Mode Seeking »
Carl Doersch · Abhinav Gupta · Alexei A Efros -
2010 Poster: Estimating Spatial Layout of Rooms using Volumetric Reasoning about Objects and Surfaces »
David C Lee · Abhinav Gupta · Martial Hebert · Takeo Kanade -
2008 Poster: A "Shape Aware" Model for semi-supervised Learning of Objects and its Context »
Abhinav Gupta · Jianbo Shi · Larry Davis -
2008 Spotlight: A "Shape Aware'' Model for semi-supervised Learning of Objects and its Context »
Abhinav Gupta · Jianbo Shi · Larry Davis