Timezone: »

 
Poster
Making Scalable Meta Learning Practical
Sang Choe · Sanket Vaibhav Mehta · Hwijeen Ahn · Willie Neiswanger · Pengtao Xie · Emma Strubell · Eric Xing

Wed Dec 13 03:00 PM -- 05:00 PM (PST) @ Great Hall & Hall B1+B2 #1113
in Poster Session 4 »
Event URL: https://github.com/leopard-ai/betty/tree/main »

Despite its flexibility to learn diverse inductive biases in machine learning programs, meta learning (i.e.,\ learning to learn) has long been recognized to suffer from poor scalability due to its tremendous compute/memory costs, training instability, and a lack of efficient distributed training support. In this work, we focus on making scalable meta learning practical by introducing SAMA, which combines advances in both implicit differentiation algorithms and systems. Specifically, SAMA is designed to flexibly support a broad range of adaptive optimizers in the base level of meta learning programs, while reducing computational burden by avoiding explicit computation of second-order gradient information, and exploiting efficient distributed training techniques implemented for first-order gradients. Evaluated on multiple large-scale meta learning benchmarks, SAMA showcases up to 1.7/4.8x increase in throughput and 2.0/3.8x decrease in memory consumption respectively on single-/multi-GPU setups compared to other baseline meta learning algorithms. Furthermore, we show that SAMA-based data optimization leads to consistent improvements in text classification accuracy with BERT and RoBERTa large language models, and achieves state-of-the-art results in both small- and large-scale data pruning on image classification tasks, demonstrating the practical applicability of scalable meta learning across language and vision domains.

Author Information

Sang Choe (Carnegie Mellon University)
Sanket Vaibhav Mehta (Carnegie Mellon University)
Sanket Vaibhav Mehta

I'm a Ph.D. candidate at the Language Technologies Institute (LTI) at the School of Computer Science, Carnegie Mellon University, and I'm advised by Emma Strubell. I'm interested in machine learning, natural language processing, and optimization with a specific focus on learning from limited labeled data, multiple tasks, and non-stationary data distributions (Lifelong Learning, Transfer Learning, Meta-Learning, Multi-Task Learning).

Hwijeen Ahn (Carnegie Mellon University)
Willie Neiswanger (Stanford / USC)
Pengtao Xie (UC San Diego)
Emma Strubell (Carnegie Mellon University)
Eric Xing (Petuum Inc.)

More from the Same Authors