Timezone: »

 
Poly-S: Analyzing and Improving Polytropon for Data-Efficient Multi-Task Learning
Lucas Page-Caccia · Edoardo Maria Ponti · Liyuan Liu · Matheus Pereira · Nicolas Le Roux · Alessandro Sordoni

@
in Transfer Learning for Natural Language Processing »
Event URL: https://openreview.net/forum?id=g8xkPAiIlG8 »

Polytropon learns a set of modular skills, which can be re-combined and fine-tuned on novel tasks with limited data. In this paper, we first investigate what makes this method successful. Specifically, we extend the evaluation benchmark to include more datasets and design a series of controlled experiments to isolate the impact of different components.We then propose a new method, Poly-S, which allows for a more fine-grained control over the combination of skills, with no additional cost in compute at inference time. We evaluate Poly-S on three multi-task NLP benchmarks, and observe improvements over strong baselines.

Author Information

Lucas Page-Caccia (McGill University)
Edoardo Maria Ponti (University of Edinburgh)
Liyuan Liu (University of Illinois, Urbana Champaign)
Matheus Pereira (Microsoft)
Nicolas Le Roux (Microsoft Research)
Alessandro Sordoni (Microsoft Research Montreal)

More from the Same Authors