Timezone: »

 
Poster
Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning
Xinyang Chen · Sinan Wang · Bo Fu · Mingsheng Long · Jianmin Wang

Wed Dec 11 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #39
in Algorithms -- Multitask and Transfer Learning »

Before sufficient training data is available, fine-tuning neural networks pre-trained on large-scale datasets substantially outperforms training from random initialization. However, fine-tuning methods suffer from a dilemma across catastrophic forgetting and negative transfer. While several methods with explicit attempts to overcome catastrophic forgetting have been proposed, negative transfer is rarely delved into. In this paper, we launch an in-depth empirical investigation into negative transfer in fine-tuning and find that, for the weight parameters and feature representations, transferability of their spectral components is diverse. For safe transfer learning, we present Batch Spectral Shrinkage (BSS), a novel regularization approach to penalizing smaller singular values so that untransferable spectral components are suppressed. BSS is orthogonal to existing fine-tuning methods and is readily pluggable into them. Experimental results show that BSS can significantly enhance the performance of state-of-the-art methods, especially in few training data regime.

Keywords: Multitask and Transfer Learning · Efficient Training Methods · Algorithms · Deep Learning
[ Paper]  [ Slides] 

Author Information

Xinyang Chen (Tsinghua University)
Sinan Wang (Tsinghua University)
Bo Fu (Tsinghua University)
Mingsheng Long (Tsinghua University)
Jianmin Wang (Tsinghua University)

More from the Same Authors