Timezone: »

 
Poster
Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve
Giannis Daras · Negin Raoof · Zoi Gkalitsiou · Alex Dimakis

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #513
in Poster Session 5 »

We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as random deletions, magnitude pruning and weight noise. Our study is motivated by research in cognitive science showing that symptoms of dementia and cognitive decline appear later in bilingual speakers compared to monolingual patients with similar brain damage, a phenomenon called bilingual cognitive reserve. Our language model experiments replicate this phenomenon on bilingual GPT-2 and other models.We provide a theoretical justification of this robustness by mathematically analyzing linear representation learning and showing that multitasking creates more robust representations. We open-source our code and models in the following URL: https://github.com/giannisdaras/multilingual_robustness.

Keywords: robustness · multitask learning · linear representation learning
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Giannis Daras (University of Texas, Austin)
Negin Raoof (University of Texas at Austin)
Zoi Gkalitsiou (University of Texas at Austin)
Alex Dimakis (University of Texas, Austin)

More from the Same Authors