Timezone: »

 
Poster
Perturbative Black Box Variational Inference
Robert Bamler · Cheng Zhang · Manfred Opper · Stephan Mandt

Wed Dec 06 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #181 #None
in Posters Wed »

Black box variational inference (BBVI) with reparameterization gradients triggered the exploration of divergence measures other than the Kullback-Leibler (KL) divergence, such as alpha divergences. These divergences can be tuned to be more mass-covering (preventing overfitting in complex models), but are also often harder to optimize using Monte-Carlo gradients. In this paper, we view BBVI with generalized divergences as a form of biased importance sampling. The choice of divergence determines a bias-variance tradeoff between the tightness of the bound (low bias) and the variance of its gradient estimators. Drawing on variational perturbation theory of statistical physics, we use these insights to construct a new variational bound which is tighter than the KL bound and more mass covering. Compared to alpha-divergences, its reparameterization gradients have a lower variance. We show in several experiments on Gaussian Processes and Variational Autoencoders that the resulting posterior covariances are closer to the true posterior and lead to higher likelihoods on held-out data.

Keywords: Variational Inference

Author Information

Robert Bamler (Disney Research)

Robert Bamler is a Postdoctoral Associate at Disney Research. He works on scalable methods for approximate Bayesian inference and on applications to natural language processing. Robert received his PhD in theoretical condensed matter physics from University of Cologne, Germany in 2016.

Cheng Zhang (Disney Research)
Manfred Opper (TU Berlin)
Stephan Mandt (Disney Research)

More from the Same Authors