Timezone: »
We propose an alternative framework to existing setups for controlling false alarms when multiple A/B tests are run over time. This setup arises in many practical applications, e.g. when pharmaceutical companies test new treatment options against control pills for different diseases, or when internet companies test their default webpages versus various alternatives over time. Our framework proposes to replace a sequence of A/B tests by a sequence of best-arm MAB instances, which can be continuously monitored by the data scientist. When interleaving the MAB tests with an an online false discovery rate (FDR) algorithm, we can obtain the best of both worlds: low sample complexity and any time online FDR control. Our main contributions are: (i) to propose reasonable definitions of a null hypothesis for MAB instances; (ii) to demonstrate how one can derive an always-valid sequential p-value that allows continuous monitoring of each MAB test; and (iii) to show that using rejection thresholds of online-FDR algorithms as the confidence levels for the MAB algorithms results in both sample-optimality, high power and low FDR at any point in time. We run extensive simulations to verify our claims, and also report results on real data collected from the New Yorker Cartoon Caption contest.
Author Information
Fanny Yang (ETH Zurich)
Aaditya Ramdas (University of California, Berkeley)
Kevin Jamieson (University of Washington)
Martin Wainwright (UC Berkeley)
Related Events (a corresponding poster, oral, or spotlight)
-
2017 Spotlight: A framework for Multi-A(rmed)/B(andit) Testing with Online FDR Control »
Wed. Dec 6th 01:35 -- 01:40 AM Room Hall C
More from the Same Authors
-
2021 : Boosting worst-group accuracy without group annotations »
Vincent Bardenhagen · Alexandru Tifrea · Fanny Yang -
2021 Poster: Interpolation can hurt robust generalization even when there is no noise »
Konstantin Donhauser · Alexandru Tifrea · Michael Aerni · Reinhard Heckel · Fanny Yang -
2020 Poster: FedSplit: an algorithmic framework for fast federated optimization »
Reese Pathak · Martin Wainwright -
2020 Poster: Preference learning along multiple criteria: A game-theoretic perspective »
Kush Bhatia · Ashwin Pananjady · Peter Bartlett · Anca Dragan · Martin Wainwright -
2018 Poster: Theoretical guarantees for EM under misspecified Gaussian mixture models »
Raaz Dwivedi · nhật Hồ · Koulik Khamaru · Martin Wainwright · Michael Jordan -
2017 Poster: Online control of the false discovery rate with decaying memory »
Aaditya Ramdas · Fanny Yang · Martin Wainwright · Michael Jordan -
2017 Poster: Early stopping for kernel boosting algorithms: A general analysis with localized complexities »
Yuting Wei · Fanny Yang · Martin Wainwright -
2017 Spotlight: Early stopping for kernel boosting algorithms: A general analysis with localized complexities »
Yuting Wei · Fanny Yang · Martin Wainwright -
2017 Oral: Online control of the false discovery rate with decaying memory »
Aaditya Ramdas · Fanny Yang · Martin Wainwright · Michael Jordan -
2016 Poster: The Power of Adaptivity in Identifying Statistical Alternatives »
Kevin Jamieson · Daniel Haas · Benjamin Recht -
2016 Poster: Finite Sample Prediction and Recovery Bounds for Ordinal Embedding »
Lalit Jain · Kevin Jamieson · Rob Nowak -
2015 Poster: NEXT: A System for Real-World Development, Evaluation, and Application of Active Learning »
Kevin G Jamieson · Lalit Jain · Chris Fernandez · Nicholas J. Glattard · Rob Nowak -
2015 Spotlight: NEXT: A System for Real-World Development, Evaluation, and Application of Active Learning »
Kevin G Jamieson · Lalit Jain · Chris Fernandez · Nicholas J. Glattard · Rob Nowak -
2012 Poster: Query Complexity of Derivative-Free Optimization »
Kevin G Jamieson · Rob Nowak · Benjamin Recht -
2011 Poster: Active Ranking using Pairwise Comparisons »
Kevin G Jamieson · Rob Nowak