Timezone: »

 
On the Role of Pre-training for Meta Few-Shot Learning
Chia-You Chen · Hsuan-Tien Lin · Masashi Sugiyama · Gang Niu

@
in 5th Workshop on Meta-Learning »
Event URL: https://openreview.net/forum?id=rMb5uMu1vuj »

Few-shot learning aims to classify unknown classes of examples with a few new examples per class. There are two key routes for few-shot learning. One is to (pre-)train a classifier with examples from known classes, and then transfer the pre-trained classifier to unknown classes using the new examples. The other, called meta few-shot learning, is to couple pre-training with episodic training, which contains episodes of few-shot learning tasks simulated from the known classes. Pre-training is known to play a crucial role for the transfer route, but the role of pre-training for the episodic route is less clear. In this work, we study the role of pre-training for the episodic route. We find that pre-training serves as major role of disentangling representations of known classes, which makes the resulting learning tasks easier for episodic training. The finding allows us to shift the huge simulation burden of episodic training to a simpler pre-training stage. We justify such a benefit of shift by designing a new disentanglement-based pre-training model, which helps episodic training achieve competitive performance more efficiently.

Author Information

Chia-You Chen (Department of computer science and informational engineering, National Taiwan University)
Hsuan-Tien Lin (National Taiwan University)
Hsuan-Tien Lin

Professor Hsuan-Tien Lin received a B.S. in Computer Science and Information Engineering from National Taiwan University in 2001, an M.S. and a Ph.D. in Computer Science from California Institute of Technology in 2005 and 2008, respectively. He joined the Department of Computer Science and Information Engineering at National Taiwan University as an assistant professor in 2008 and has been promoted to full professor in 2017. Between 2016 and 2019, he worked as the Chief Data Scientist of Appier, a startup company that specializes in making AI easier for marketing. Currently, he keeps growing with Appier as its Chief Data Science Consultant. From the university, Prof. Lin received the Distinguished Teaching Awards in 2011 and 2021, the Outstanding Mentoring Award in 2013, and five Outstanding Teaching Awards between 2016 and 2020. He co-authored the introductory machine learning textbook Learning from Data and offered two popular Mandarin-teaching MOOCs Machine Learning Foundations and Machine Learning Techniques based on the textbook. He served in the machine learning community as Progam Co-chair of NeurIPS 2020, Expo Co-chair of ICML 2021, and Workshop Chair of NeurIPS 2022 and 2023. He co-led the teams that won the champion of KDDCup 2010, the double-champion of the two tracks in KDDCup 2011, the champion of track 2 in KDDCup 2012, and the double-champion of the two tracks in KDDCup 2013.

Masashi Sugiyama (RIKEN / University of Tokyo)
Gang Niu (RIKEN)
Gang Niu

Gang Niu is currently an indefinite-term senior research scientist at RIKEN Center for Advanced Intelligence Project.

More from the Same Authors