Timezone: »
When used to learn high dimensional parametric probabilistic models, the clas- sical maximum likelihood (ML) learning often suffers from computational in- tractability, which motivates the active developments of non-ML learning meth- ods. Yet, because of their divergent motivations and forms, the objective func- tions of many non-ML learning methods are seemingly unrelated, and there lacks a unified framework to understand them. In this work, based on an information geometric view of parametric learning, we introduce a general non-ML learning principle termed as minimum KL contraction, where we seek optimal parameters that minimizes the contraction of the KL divergence between the two distributions after they are transformed with a KL contraction operator. We then show that the objective functions of several important or recently developed non-ML learn- ing methods, including contrastive divergence [12], noise-contrastive estimation [11], partial likelihood [7], non-local contrastive objectives [31], score match- ing [14], pseudo-likelihood [3], maximum conditional likelihood [17], maximum mutual information [2], maximum marginal likelihood [9], and conditional and marginal composite likelihood [24], can be unified under the minimum KL con- traction framework with different choices of the KL contraction operators.
Author Information
Siwei Lyu (University at Albany SUNY)
More from the Same Authors
-
2016 Poster: Stochastic Online AUC Maximization »
Yiming Ying · Longyin Wen · Siwei Lyu -
2016 Oral: Stochastic Online AUC Maximization »
Yiming Ying · Longyin Wen · Siwei Lyu -
2012 Poster: Learning with Target Prior »
Zuoguan Wang · Siwei Lyu · Gerwin Schalk · Qiang Ji -
2010 Poster: Divisive Normalization: Justification and Effectiveness as Efficient Coding Transform »
Siwei Lyu -
2008 Oral: Reducing statistical dependencies in natural signals using radial Gaussianization »
Siwei Lyu · Eero Simoncelli -
2008 Poster: Reducing statistical dependencies in natural signals using radial Gaussianization »
Siwei Lyu · Eero Simoncelli -
2006 Poster: Statistical Modeling of Images with Fields of Gaussian Scale Mixtures »
Siwei Lyu · Eero Simoncelli