Timezone: »

Kernel Identification Through Transformers
Fergus Simpson · Ian Davies · Vidhi Lalchand · Alessandro Vullo · Nicolas Durrande · Carl Edward Rasmussen

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @ None #None

Kernel selection plays a central role in determining the performance of Gaussian Process (GP) models, as the chosen kernel determines both the inductive biases and prior support of functions under the GP prior. This work addresses the challenge of constructing custom kernel functions for high-dimensional GP regression models. Drawing inspiration from recent progress in deep learning, we introduce a novel approach named KITT: Kernel Identification Through Transformers. KITT exploits a transformer-based architecture to generate kernel recommendations in under 0.1 seconds, which is several orders of magnitude faster than conventional kernel search algorithms. We train our model using synthetic data generated from priors over a vocabulary of known kernels. By exploiting the nature of the self-attention mechanism, KITT is able to process datasets with inputs of arbitrary dimension. We demonstrate that kernels chosen by KITT yield strong performance over a diverse collection of regression benchmarks.

Author Information

Fergus Simpson (Secondmind)
Ian Davies (InstaDeep)
Vidhi Lalchand (University of Cambridge)

Ph.D student in Machine learning at Cambridge, I work on Bayesian Non-parametrics, Gaussian Processes, Kernel Learning. Application Areas: High Energy Physics, Astronomy, Science!

Alessandro Vullo (University College Dublin)
Nicolas Durrande (Secondmind)
Carl Edward Rasmussen (University of Cambridge)

More from the Same Authors