Timezone: »

 
FiT: Parameter Efficient Few-shot Transfer Learning
Aliaksandra Shysheya · John Bronskill · Massimiliano Patacchiola · Sebastian Nowozin · Richard Turner

@
in NeurIPS 2022 Workshop on Meta-Learning »
Event URL: https://openreview.net/forum?id=F3N4XrLCCm »

Model parameter efficiency is key for enabling few-shot learning, inexpensive model updates for personalization, and communication efficient federated learning. In this work, we develop FiLM Transfer (FiT) which combines ideas from transfer learning (fixed pretrained backbones and fine-tuned FiLM adapter layers) and meta-learning (automatically configured Naive Bayes classifiers and episodic training) to yield parameter efficient models with superior classification accuracy at low-shot. We experiment with FiT on a range of downstream datasets and show that it achieves better classification accuracy than the leading Big Transfer (BiT) algorithm at low-shot and achieves state-of-the art accuracy on the challenging VTAB-1k benchmark, with fewer than 1% of the updateable parameters.

Author Information

Aliaksandra Shysheya (University of Cambridge)
John Bronskill (University of Cambridge)
Massimiliano Patacchiola (University of Cambridge)
Massimiliano Patacchiola

Massimiliano (Max) Patacchiola is a postdoctoral researcher at the University of Cambridge (Machine Learning Group) working under the supervision of prof. Richard Turner in collaboration with Microsoft Research. Before he was a postdoctoral researcher at the University of Edinburgh and an inter at Snapchat. Max is interested in meta-learning, few-shot learning, and reinforcement learning.

Sebastian Nowozin (DeepMind)
Richard Turner (University of Cambridge)

More from the Same Authors