Timezone: »

 
Poster
R-Drop: Regularized Dropout for Neural Networks
xiaobo liang · Lijun Wu · Juntao Li · Yue Wang · Qi Meng · Tao Qin · Wei Chen · Min Zhang · Tie-Yan Liu

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @
in Poster Session 1 »
Dropout is a powerful and widely used technique to regularize the training of deep neural networks. Though effective and performing well, the randomness introduced by dropout causes unnegligible inconsistency between training and inference. In this paper, we introduce a simple consistency training strategy to regularize dropout, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other. Specifically, for each training sample, R-Drop minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by dropout. Theoretical analysis reveals that R-Drop reduces the above inconsistency. Experiments on $\bf{5}$ widely used deep learning tasks ($\bf{18}$ datasets in total), including neural machine translation, abstractive summarization, language understanding, language modeling, and image classification, show that R-Drop is universally effective. In particular, it yields substantial improvements when applied to fine-tune large-scale pre-trained models, e.g., ViT, RoBERTa-large, and BART, and achieves state-of-the-art (SOTA) performances with the vanilla Transformer model on WMT14 English$\to$German translation ($\bf{30.91}$ BLEU) and WMT14 English$\to$French translation ($\bf{43.95}$ BLEU), even surpassing models trained with extra large-scale data and expert-designed advanced variants of Transformer models. Our code is available at GitHub\footnote{\url{https://github.com/dropreg/R-Drop}}.
Keywords: Deep Learning · Machine Learning · Transformers · Vision · Language

Author Information

xiaobo liang (Soochow University, China)
Lijun Wu (Sun Yat-sen University)
Juntao Li (Soochow University, China)
Yue Wang (Soochow University, China)
Qi Meng (Microsoft)
Tao Qin (Microsoft Research)
Wei Chen (Chinese Academy of Sciences)
Min Zhang
Tie-Yan Liu (Microsoft Research Asia)

More from the Same Authors