Timezone: »
In this paper we explore the problem of biasing unsupervised models to favor sparsity. We extend the posterior regularization framework [8] to encourage the model to achieve posterior sparsity on the unlabeled training data. We apply this new method to learn first-order HMMs for unsupervised part-of-speech (POS) tagging, and show that HMMs learned this way consistently and significantly out-performs both EM-trained HMMs, and HMMs with a sparsity-inducing Dirichlet prior trained by variational EM. We evaluate these HMMs on three languages — English, Bulgarian and Portuguese — under four conditions. We find that our method always improves performance with respect to both baselines, while variational Bayes actually degrades performance in most cases. We increase accuracy with respect to EM by 2.5%-8.7% absolute and we see improvements even in a semisupervised condition where a limited dictionary is provided.
Author Information
Joao V Graca (L2F INESC-ID Lisboa)
Kuzman Ganchev (University of Pennsylvania)
Ben Taskar (University of Washington)
Fernando Pereira (Google)
Related Events (a corresponding poster, oral, or spotlight)
-
2009 Spotlight: Posterior vs Parameter Sparsity in Latent Variable Models »
Thu. Dec 10th 01:23 -- 01:24 AM Room
More from the Same Authors
-
2020 Poster: Faithful Embeddings for Knowledge Base Queries »
Haitian Sun · Andrew Arnold · Tania Bedrax Weiss · Fernando Pereira · William Cohen -
2014 Poster: Expectation-Maximization for Learning Determinantal Point Processes »
Jennifer A Gillenwater · Alex Kulesza · Emily Fox · Ben Taskar -
2013 Poster: Learning Adaptive Value of Information for Structured Prediction »
David J Weiss · Ben Taskar -
2013 Poster: Approximate Inference in Continuous Determinantal Processes »
Raja Hafiz Affandi · Emily Fox · Ben Taskar -
2013 Spotlight: Approximate Inference in Continuous Determinantal Processes »
Raja Hafiz Affandi · Emily Fox · Ben Taskar -
2012 Poster: Near-Optimal MAP Inference for Determinantal Point Processes »
Alex Kulesza · Jennifer A Gillenwater · Ben Taskar -
2012 Oral: Near-Optimal MAP Inference for Determinantal Point Processes »
Alex Kulesza · Jennifer A Gillenwater · Ben Taskar -
2011 Session: Opening Remarks and Awards »
Terrence Sejnowski · Peter Bartlett · Fernando Pereira -
2010 Workshop: Coarse-to-Fine Learning and Inference »
Ben Taskar · David J Weiss · Benjamin J Sapp · Slav Petrov -
2010 Spotlight: Structured Determinantal Point Processes »
Alex Kulesza · Ben Taskar -
2010 Poster: Structured Determinantal Point Processes »
Alex Kulesza · Ben Taskar -
2010 Oral: Semi-Supervised Learning with Adversarially Missing Label Information »
Umar Syed · Ben Taskar -
2010 Session: Spotlights Session 3 »
Ben Taskar -
2010 Session: Oral Session 3 »
Ben Taskar -
2010 Poster: Semi-Supervised Learning with Adversarially Missing Label Information »
Umar Syed · Ben Taskar -
2010 Poster: Sidestepping Intractable Inference with Structured Ensemble Cascades »
David J Weiss · Benjamin J Sapp · Ben Taskar -
2009 Session: Oral Session 6: Theory, Optimization and Games »
Ben Taskar -
2009 Poster: Group Sparse Coding »
Samy Bengio · Fernando Pereira · Yoram Singer · Dennis Strelow -
2008 Poster: Exact Convex Confidence-Weighted Learning »
Yacov Crammer · Mark Dredze · Fernando Pereira -
2008 Spotlight: Exact Convex Confidence-Weighted Learning »
Yacov Crammer · Mark Dredze · Fernando Pereira -
2007 Poster: Expectation Maximization, Posterior Constraints, and Statistical Alignment »
Kuzman Ganchev · Joao V Graca · Ben Taskar -
2007 Spotlight: Expectation Maximization, Posterior Constraints, and Statistical Alignment »
Kuzman Ganchev · Joao V Graca · Ben Taskar -
2007 Spotlight: Structured Learning with Approximate Inference »
Alex Kulesza · Fernando Pereira -
2007 Tutorial: Structured Prediction »
Ben Taskar -
2007 Poster: Structured Learning with Approximate Inference »
Alex Kulesza · Fernando Pereira -
2007 Poster: Learning Bounds for Domain Adaptation »
John Blitzer · Yacov Crammer · Alex Kulesza · Fernando Pereira · Jennifer Wortman Vaughan -
2006 Poster: Analysis of Representations for Domain Adaptation »
John Blitzer · Shai Ben-David · Yacov Crammer · Fernando Pereira