Timezone: »

 
Robust Reinforcement Learning for Shifting Dynamics During Deployment
Samuel Stanton · Rasool Fakoor · Jonas Mueller · Andrew Gordon Wilson · Alexander Smola

@
in Safe and Robust Control of Uncertain Systems »

While high-return policies can be learned on a wide range of systems through reinforcement learning, actual deployment of the resulting policies is often hindered by their sensitivity to future changes in the environment. Adversarial training has shown some promise in producing policies that retain better performance under environment shifts, but existing approaches only consider robustness to specific kinds of perturbations that must be specified a priori. As possible changes in future dynamics are typically unknown in practice, we instead seek a policy that is robust to a variety of realistic changes only encountered at test-time. Towards this goal, we propose a new adversarial variant of soft actor-critic, which produces policies on Mujoco continuous control tasks that are simultaneously more robust across various environment shifts, such as changes to friction and body mass.

Author Information

Samuel Stanton (New York University)

Sam is a Ph.D. student in the NYU Center for Data Science and a NDSEG Fellow (class of 2018), working with Professor Andrew Wilson. His current research focuses on the incorporation of probabilistic state transition models in reinforcement learning algorithms. Model-based RL agents generalize from past experience very effectively, allowing the agent to evaluate policies with fewer environment interactions than their model-free counterparts. Improving the data-efficiency of RL agents is crucial for real-world applications in fields like robotics, logistics, and finance. Sam holds a Master’s degree in Operations Research from Cornell University, where he started working with Professor Wilson as a first-year Ph.D. student. Sam transferred from the Cornell doctoral program to continue his research agenda at NYU with his advisor. Prior to his studies at Cornell, Sam earned a Bachelor’s degree in Mathematics from the University of Colorado Denver, graduating summa cum laude. In addition to his dissertation research, Sam is interested in modern art and philosophy, especially epistemology and ethics. When he is not occupied with research, Sam enjoys volleyball, rock climbing, surfing, and snowboarding.

Rasool Fakoor (Amazon Web Services)
Jonas Mueller (Amazon Web Services)
Andrew Gordon Wilson (New York University)
Alexander Smola (Amazon)

More from the Same Authors