Bet Imaginatively, not Historically in Independent-Data Sequential Testing

Sequential testing avoids some of the many controversies and drawbacks related to traditional batch testing with p-values. In particular, the testing by betting paradigm has developed rapidly in recent years and frames the level of evidence against the null as the wealth of a player in a betting game. Naturally, how much a player should bet is a key consideration, and should reflect the statistician's confidence in what they know about the next observed data point. Much of current work uses standard betting strategies developed for adversarial problems, where the data can be hand-picked to the detriment of the player. In the stochastic setting, where the data observed is independent and stationary, these approaches may be suboptimal. We propose a simple betting strategy, **follow the leave-one-out e-power (FLOE), that exploits the i.i.d. nature of a data stream to estimate the e-power of our current test on the underlying distribution. We average not over the historical evidence collected so far, as generic online learning-based methods do, but instead over the equally-likely possible data sequences we might have seen based on shuffling the data. With an additional principled regularization technique, this simple drop-in method yields substantial improvements in testing performance on a variety of kernel two-sample testing problems.