Timezone: »

 
Are you using test log-likelihood correctly?
Sameer Deshpande · Soumya Ghosh · Tin Nguyen · Tamara Broderick

@
in I Can’t Believe It’s Not Better: Understanding Deep Learning Through Empirical Falsification »
Event URL: https://openreview.net/forum?id=6fHJniMUra »

Test log-likelihood is commonly used to compare different models of the same data and different approximate inference algorithms for fitting the same probabilistic model. We present simple examples demonstrating how comparisons based on test log-likelihood can contradict comparisons according to other objectives. Specifically, our examples show that (i) conclusions about forecast accuracy based on test log-likelihood comparisons may not agree with conclusions based on other distributional quantities like means; and (ii) that approximate Bayesian inference algorithms that attain higher test log-likelihoods need not also yield more accurate posterior approximations.

Author Information

Sameer Deshpande (University of Wisconsin--Madison)
Soumya Ghosh (MIT-IBM Watson AI Lab, IBM Research)
Tin Nguyen (MIT)
Tamara Broderick (MIT)

More from the Same Authors