Timezone: »

 
Poster
Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation
Zhiwei Hao · Jianyuan Guo · Ding Jia · Kai Han · Yehui Tang · Chao Zhang · Han Hu · Yunhe Wang

@

In the past few years, transformers have achieved promising performance on various computer vision tasks. Unfortunately, the immense inference overhead of most existing vision transformers withholds them from being deployed on edge devices such as cell phones and smart watches. Knowledge distillation is a widely used paradigm for compressing cumbersome architectures into compact students via transferring information. However, most of them are designed for convolutional neural networks (CNNs), which do not fully investigate the character of vision transformers. In this paper, we fully utilize the patch-level information and propose a fine-grained manifold distillation method for transformer-based networks. Specifically, we train a tiny student model to match a pre-trained teacher model in the patch-level manifold space. Then, we decouple the manifold matching loss into three terms with careful design to further reduce the computational costs for the patch relationship. Equipped with the proposed method, a DeiT-Tiny model containing 5M parameters achieves 76.5\% top-1 accuracy on ImageNet-1k, which is +2.0\% higher than previous distillation approaches. Transfer learning results on other classification benchmarks and downstream vision tasks also demonstrate the superiority of our method over the state-of-the-art algorithms.

Keywords: Manifold Learning · Vision transformer · knowledge diatillation
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Zhiwei Hao (Beijing Institute of Technology)
Jianyuan Guo (University of Sydney)
Ding Jia (Peking University)
Kai Han (Huawei Noah's Ark Lab)
Yehui Tang (Peking University)
Chao Zhang (Peking University)
Han Hu (Beijing Institute of Technology)
Yunhe Wang (Huawei Noah's Ark Lab)

More from the Same Authors