Timezone: »

 
One-Shot Optimal Design for Gaussian Process Analysis of Randomized Experiments
Jelena Markovic · Qing Feng · Eytan Bakshy

@
in NeurIPS 2022 Workshop on Meta-Learning »
Event URL: https://openreview.net/forum?id=FAcfILpleA »

Bayesian Optimization provides a sample-efficient approach to optimize Internet systems that are evaluated with randomized experiments. Such evaluations are often resource- and time- consuming in order to measure noisy and long-term outcomes. Thus, the initial randomized design, i.e. determining number of test groups and sample sizes, plays a critical role in building an accurate Gaussian Process model to optimize efficiently and decreasing experimentation cost. We develop a simulation-based method with meta-learned priors to decide the optimal design for the initial batch of GP-modeled randomized experiments. The meta-learning is performed on a large corpus of randomized experiments conducted at Meta and obtains sensible GP priors for simulating across different designs. The one-shot optimal design policy is derived by training a machine learning model with simulation data to map experiment characteristics to an optimal design. Our evaluations show that our proposed optimal design significantly improves resource-efficiency while achieving a target GP model accuracy.

Author Information

Jelena Markovic (Meta)
Qing Feng (Facebook)
Eytan Bakshy (Meta)

More from the Same Authors