Timezone: »
Bandit algorithms are increasingly used in real-world sequential decision-making problems. Associated with this is an increased desire to be able to use the resulting datasets to answer scientific questions like: Did one type of ad lead to more purchases? In which contexts is a mobile health intervention effective? However, classical statistical approaches fail to provide valid confidence intervals when used with data collected with bandit algorithms. Alternative methods have recently been developed for simple models (e.g., comparison of means). Yet there is a lack of general methods for conducting statistical inference using more complex models on data collected with (contextual) bandit algorithms; for example, current methods cannot be used for valid inference on parameters in a logistic regression model for a binary reward. In this work, we develop theory justifying the use of M-estimators---which includes estimators based on empirical risk minimization as well as maximum likelihood---on data collected with adaptive algorithms, including (contextual) bandit algorithms. Specifically, we show that M-estimators, modified with particular adaptive weights, can be used to construct asymptotically valid confidence regions for a variety of inferential targets.
Author Information
Kelly Zhang (Harvard University)
Lucas Janson (Harvard University)
Susan Murphy (Harvard University)
Susan A. Murphy is Professor of Statistics and Computer Science at Harvard University. Her research focuses on improving sequential decision making in health, in particular the development of online, real-time reinforcement learning algorithms for use in personalized digital health. She is a member of the US National Academy of Sciences and of the US National Academy of Medicine. In 2013 she was awarded a MacArthur Fellowship for her work on experimental designs to inform sequential decision making. She is a Fellow of the College on Problems in Drug Dependence, Past-President of Institute of Mathematical Statistics, and a former editor of the Annals of Statistics.
More from the Same Authors
-
2021 Workshop: Causal Inference Challenges in Sequential Decision Making: Bridging Theory and Practice »
Aurelien Bibaut · Maria Dimakopoulou · Nathan Kallus · Xinkun Nie · Masatoshi Uehara · Kelly Zhang -
2020 Workshop: Machine Learning for Mobile Health »
Joseph Futoma · Walter Dempsey · Katherine Heller · Yian Ma · Nicholas Foti · Marianne Njifon · Kelly Zhang · Jieru Shi -
2020 : Panel discussion »
Pierre-Yves Oudeyer · Marc Bellemare · Peter Stone · Matt Botvinick · Susan Murphy · Anusha Nagabandi · Ashley Edwards · Karen Liu · Pieter Abbeel -
2020 : Invited talk: Susan Murphy "We used RL but…. Did it work?!" »
Susan Murphy -
2020 Poster: Cross-validation Confidence Intervals for Test Error »
Pierre Bayle · Alexandre Bayle · Lucas Janson · Lester Mackey -
2020 Poster: Inference for Batched Bandits »
Kelly Zhang · Lucas Janson · Susan Murphy -
2017 : Keynote: Susan Murphy, U. Michigan »
Susan Murphy -
2017 Poster: Action Centered Contextual Bandits »
Kristjan Greenewald · Ambuj Tewari · Susan Murphy · Predag Klasnja -
2010 Workshop: Learning and Planning from Batch Time Series Data »
Daniel Lizotte · Michael Bowling · Susan Murphy · Joelle Pineau · Sandeep Vijan