Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization

Gaussian Process (GP) kernels are central to Bayesian optimization (BO), yet designing effective kernels for high-dimensional problems still relies on extensive manual engineering. Existing kernel design methods restrict the search to additive and multiplicative compositions of base kernels, while LLM-based BO approaches condition on raw observations, which are infeasible in high dimensions due to context-length constraints and the difficulty of extracting meaningful patterns from high-dimensional numerical observations. We introduce Kernel Discovery, an LLM-driven population-based evolutionary framework that overcomes both limitations. Motivated by the observation that directly prompting an LLM to generate kernel code yields syntactically varied but functionally identical kernels, we adopt a two-stage approach: an LLM first proposes novel mathematical forms, then a second LLM call converts each form into validated, executable code. We also propose a leave-one-out continuous ranked probability score (LOO-CRPS) as a held-out predictive scoring criterion that penalizes overconfident fits more directly than marginal log-likelihood. On five standard high-dimensional BO benchmarks, our method achieves an average rank of 1.2 out of 17, consistently outperforming competitive baselines. We further analyze the discovered kernels to examine which kernel characteristics lead to superior performance in high-dimensional BO.