One-Shot Optimal Design for Gaussian Process Analysis of Randomized Experiments

Abstract

Bayesian optimization provides a sample-efficient approach to optimize systems that are evaluated with randomized experiments, such as Internet experiments (A/B tests) and clinical trials. Such evaluations are often resource- and time-consuming in order to measure noisy and long-term outcomes. Thus, the initial randomized design, i.e., determining the number of test groups and their sample sizes, plays a critical role in building an accurate Gaussian Process (GP) model to optimize efficiently and decreasing experimentation cost. We develop a simulation-based method with meta-learned priors to decide the optimal design for the initial batch of GP-modeled randomized experiments. The meta-learning is performed on a large corpus of randomized experiments conducted at Meta, obtaining sensible GP priors for simulating across different designs. The one-shot optimal design policy is derived by training a machine learning model with simulation data to map experiment characteristics to an optimal design. Our evaluations show that our proposed optimal design significantly improves resource-efficiency while achieving a target GP model accuracy.