Timezone: »

 
Towards Data-Driven Offline Simulations for Online Reinforcement Learning
Shengpu Tang · Felipe Vieira Frujeri · Dipendra Misra · Alex Lamb · John Langford · Paul Mineiro · Sebastian Kochman

@
in 3rd Offline Reinforcement Learning Workshop: Offline RL as a "Launchpad" »
Event URL: https://openreview.net/forum?id=p5VJFCoR59 »

Modern decision-making systems, from robots to web recommendation engines, are expected to adapt: to user preferences, changing circumstances or even new tasks. Yet, it is still uncommon to deploy a dynamically learning agent (rather than a fixed policy) to a production system, as it's perceived as unsafe. Using historical data to reason about learning algorithms, similar to offline policy evaluation (OPE) applied to fixed policies, could help practitioners evaluate and ultimately deploy such adaptive agents to production. In this work, we formalize offline learner simulation (OLS) for reinforcement learning (RL) and propose a novel evaluation protocol that measures both fidelity and efficiency. For environments with complex high-dimensional observations, we propose a semi-parametric approach that leverages recent advances in latent state discovery. In preliminary experiments, we show the advantage of our approach compared to fully non-parametric baselines.

Author Information

Shengpu Tang (University of Michigan)
Felipe Vieira Frujeri (Microsoft Research)
Dipendra Misra
Alex Lamb (Universite de Montreal)
John Langford (Microsoft Research)

John Langford is a machine learning research scientist, a field which he says "is shifting from an academic discipline to an industrial tool". He is the author of the weblog hunch.net and the principal developer of Vowpal Wabbit. John works at Microsoft Research New York, of which he was one of the founding members, and was previously affiliated with Yahoo! Research, Toyota Technological Institute, and IBM's Watson Research Center. He studied Physics and Computer Science at the California Institute of Technology, earning a double bachelor's degree in 1997, and received his Ph.D. in Computer Science from Carnegie Mellon University in 2002. He was the program co-chair for the 2012 International Conference on Machine Learning.

Paul Mineiro (Microsoft)
Sebastian Kochman (Technical University of Poznan)

More from the Same Authors