Program Highlights »
Sat Dec 9th 08:00 AM -- 06:30 PM @ Grand Ballroom B
Learning with Limited Labeled Data: Weak Supervision and Beyond
Isabelle Augenstein · Stephen Bach · Eugene Belilovsky · Matthew Blaschko · Christoph Lampert · Edouard Oyallon · Emmanouil Antonios Platanios · Alexander Ratner · Christopher Ré

Workshop Home Page

Modern representation learning techniques like deep neural networks have had a major impact both within and beyond the field of machine learning, achieving new state-of-the-art performances with little or no feature engineering on a vast array of tasks. However, these gains are often difficult to translate into real-world settings as they require massive hand-labeled training sets. And in the vast majority of real-world settings, collecting such training sets by hand is infeasible due to the cost of labeling data or the paucity of data in a given domain (e.g. rare diseases in medical applications). In this workshop we focus on techniques for few sample learning and using weaker supervision when large unlabeled datasets are available, as well as theory associated with both.

One increasingly popular approach is to use weaker forms of supervision—i.e. supervision that is potentially noisier, biased, and/or less precise. An overarching goal of such approaches is to use domain knowledge and resources from subject matter experts, but to solicit it in higher-level, lower-fidelity, or more opportunistic ways. Examples include higher-level abstractions such as heuristic labeling rules, feature annotations, constraints, expected distributions, and generalized expectation criteria; noisier or biased labels from distant supervision, crowd workers, and weak classifiers; data augmentation strategies to express class invariances; and potentially mismatched training data such as in multitask and transfer learning settings.

Along with practical methods and techniques for dealing with limited labeled data settings, this workshop will also focus on the theory of learning in this general setting. Although several classic techniques in the statistical learning theory exist which handle the case of few samples and high dimensions, extending these results for example to the recent success of deep learning is still a challenge. How can the theory or the techniques that have gained success in deep learning be adapted to the case of limited labeled data? How can systems designed (and potentially deployed) for large scale learning be adapted to small data settings? What are efficient and practical ways to incorporate prior knowledge?

This workshop will focus on highlighting both practical and theoretical aspects of learning with limited labeled data, including but not limited to topics such as:
- Learning from noisy labels
- “Distant” or heuristic supervision
- Non-standard labels such as feature annotations, distributions, and constraints
-Data augmentation and/or the use of simulated data
- Frameworks that can tackle both very few samples and settings with more data without
extensive intervention.
- Effective and practical techniques for incorporating domain knowledge
- Applications of machine learning for small data problems in medical images and industry

08:30 AM Welcome & Opening Remarks (Talk)
08:40 AM Invited Talk: "Tales from fMRI: Learning from limited labeled data" (Invited Talk)
Gael Varoquaux
09:10 AM Invited Talk: Learning from Limited Labeled Data (But a Lot of Unlabeled Data) (Invited Talk)
Tom Mitchell
09:40 AM Contributed Talk 1: "Smooth Neighbors on Teacher Graphs for Semi-supervised Learning" (Contributed Talk)
09:55 AM 1-minute Poster Spotlights (Session #1) (Spotlights)
10:15 AM Poster Sessions (Poster Session)
Dennis Forster, David Inouye, Shashank Srivastava, Martine De Cock, Srinagesh Sharma, Mateusz Kozinski, Petr Babkin, maxime he, Zhe Cui, Shivani Rao, Ramesh Raskar, Pradipto Das, Albert Zhao, Ravi Lanka
11:00 AM Invited Talk: "Light Supervision of Structured Prediction Energy Networks" (Invited Talk)
Andrew McCallum
11:30 AM Invited Talk: "Forcing Neural Link Predictors to Play by the Rules", Sebastian Riedel (Invited Talk)
12:00 PM Lunch
02:00 PM Panel: Limited Labeled Data in Medical Imaging (Panel)
Daniel Rubin, Matthew Lungren
02:30 PM 1-minute Poster Spotlights (Session #2) (Spotlights)
02:50 PM Poster Session / Coffee Break (Poster Session)
Hongyu Ren, Sheng Lundquist, Steven Hickson, Abhimanyu Dubey, Saki Shinoda, Ana Marasovic, Otilia Stretcu, Esube T Bekele, Vikas Raunak, Cicero dos Santos, Liane Canas, Jesus Mager Hois, Martin Hirzel
03:30 PM Invited Talk: Sample and Computationally Efficient Active Learning Algorithms (Invited Talk)
Maria-Florina Balcan
04:00 PM Contributed Talk 2: "EZLearn: Exploiting Organic Supervision in Large-Scale Data Annotation" (Contributed Talk)
04:15 PM Invited Talk: Overcoming Limited Data with GANs (Invited Talk)
Ian Goodfellow
04:45 PM Invited Talk: Sameer Singh, "That Doesn't Make Sense! A Case Study in Actively Annotating Model Explanations" (Invited Talk)
05:15 PM Contributed Talk 3: Local Affine Approximators of Deep Neural Nets for Improving Knowledge Transfer (Contributed Talk)
05:30 PM Contributed Talk 4: Co-trained Ensemble Models for Weakly Supervised Cyberbullying Detection (Contributed Talk)
05:45 PM Invited Talk: What’s so Hard About Natural Language Understanding? (Invited Talk)
Alan Ritter
06:15 PM Closing Remarks & Awards (Talk)