Timezone: »

 
Oral Poster
Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent
Jihao Andreas Lin · Javier Antorán · Shreyas Padhy · David Janz · José Miguel Hernández-Lobato · Alexander Terenin

Thu Dec 14 08:45 AM -- 10:45 AM (PST) @ Great Hall & Hall B1+B2 #1221
in Poster Session 5 »

Gaussian processes are a powerful framework for quantifying uncertainty and for sequential decision-making but are limited by the requirement of solving linear systems. In general, this has a cubic cost in dataset size and is sensitive to conditioning. We explore stochastic gradient algorithms as a computationally efficient method of approximately solving these linear systems: we develop low-variance optimization objectives for sampling from the posterior and extend these to inducing points. Counterintuitively, stochastic gradient descent often produces accurate predictions, even in cases where it does not converge quickly to the optimum. We explain this through a spectral characterization of the implicit bias from non-convergence. We show that stochastic gradient descent produces predictive distributions close to the true posterior both in regions with sufficient data coverage, and in regions sufficiently far away from the data. Experimentally, stochastic gradient descent achieves state-of-the-art performance on sufficiently large-scale or ill-conditioned regression tasks. Its uncertainty estimates match the performance of significantly more expensive baselines on a large-scale Bayesian~optimization~task.

Author Information

Jihao Andreas Lin (TU Darmstadt)
Javier Antorán (University of Cambridge)
Shreyas Padhy (University of Cambridge)
David Janz (University of Cambridge)
José Miguel Hernández-Lobato (University of Cambridge)
Alexander Terenin (Cornell University)
Alexander Terenin

Alexander Terenin is an Assistant Research Professor at Cornell. He is interested in machine learning, particularly for problems where the data is not fixed, but is gathered interactively by the learning machine. His work focuses on data-efficient interactive decision-making algorithms such as Bayesian optimization, and uncertainty-aware probabilistic models that power such algorithms, including Gaussian processes. His technical contributions to this area have won multiple best-paper-type awards at top machine learning conferences.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors