Deferred Aggregation in Hierarchical Bayesian Optimization

To account for uncertainty, hierarchical Bayesian optimization marginalizes over Gaussian process hyperparameters, usually averaging models or acquisition functions. However, this early aggregation discards information about model disagreement and is sensitive to outliers. To address these limitations, we propose to defer aggregation to the decision level: each model independently proposes a candidate, and we aggregate over these candidates via the medoid. We study a misspecified hierarchical Gaussian process setting in which most models have low misspecification error, while a minority are severely misspecified. Theoretically, we derive regret bounds showing that the misspecification penalty of early aggregation depends on the average error across all models, whereas decision-level aggregation depends only on the low-misspecification majority. Empirically, we validate these findings on toy examples and further demonstrate that, across a wide range of low- to high-dimensional synthetic and real-world benchmarks, decision-level aggregation outperforms standard Bayesian optimization baselines in most cases while remaining competitive in the others.