Timezone: »

 
Poster
Reusing Pretrained Models by Multi-linear Operators for Efficient Training
Yu Pan · Ye Yuan · Yichun Yin · Zenglin Xu · Lifeng Shang · Xin Jiang · Qun Liu

Wed Dec 13 03:00 PM -- 05:00 PM (PST) @ Great Hall & Hall B1+B2 #516
in Poster Session 4 »

Training large models from scratch usually costs a substantial amount of resources. Towards this problem, recent studies such as bert2BERT and LiGO have reused small pretrained models to initialize a large model (termed the ``target model''), leading to a considerable acceleration in training. Despite the successes of these previous studies, they grew pretrained models by mapping partial weights only, ignoring potential correlations across the entire model. As we show in this paper, there are inter- and intra-interactions among the weights of both the pretrained and the target models. As a result, the partial mapping may not capture the complete information and lead to inadequate growth. In this paper, we propose a method that linearly correlates each weight of the target model to all the weights of the pretrained model to further enhance acceleration ability. We utilize multi-linear operators to reduce computational and spacial complexity, enabling acceptable resource requirements. Experiments demonstrate that our method can save 76\% computational costs on DeiT-base transferred from DeiT-small, which outperforms bert2BERT by +12\% and LiGO by +21\%, respectively.

Author Information

Yu Pan (Harbin Institute of Technology, Shenzhen)
Ye Yuan (Peking University)
Yichun Yin (Huawei Noah's Ark Lab)
Zenglin Xu (Harbin Institute of Technology Shenzhen)
Lifeng Shang (Huawei Technologies Ltd.)
Xin Jiang (Noah’s Ark Lab, Huawei Technologies)
Qun Liu (Huawei Noah's Ark Lab)

More from the Same Authors