Timezone: »

 
Poster
Post-Training Quantization for Vision Transformer
Zhenhua Liu · Yunhe Wang · Kai Han · Wei Zhang · Siwei Ma · Wen Gao

Wed Dec 08 12:30 AM -- 02:00 AM (PST) @
in Poster Session 3 »

Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the functionality of the attention mechanism, we introduce a ranking loss into the conventional quantization objective that aims to keep the relative order of the self-attention results after quantization. Moreover, we thoroughly analyze the relationship between quantization loss of different layers and the feature diversity, and explore a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-art post-training quantization algorithms. For instance, we can obtain an 81.29% top-1 accuracy using DeiT-B model on ImageNet dataset with about 8-bit quantization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/VT-PTQ.

Keywords: Deep Learning · Transformers · Vision

Author Information

Zhenhua Liu (Peking University)
Yunhe Wang (Huawei Noah's Ark Lab)
Kai Han (Huawei Noah's Ark Lab)
Wei Zhang (Noah's Ark Lab, Huawei Inc.)
Siwei Ma (Peking University)
Wen Gao (peking university)

More from the Same Authors