Timezone: »

 
Poster
Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning
Weicong Liang · YUHUI YUAN · Henghui Ding · Xiao Luo · Weihong Lin · Ding Jia · Zheng Zhang · Chao Zhang · Han Hu

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #139
in Poster Session 4 »
Vision transformers have recently achieved competitive results across various vision tasks but still suffer from heavy computation costs when processing a large number of tokens. Many advanced approaches have been developed to reduce the total number of tokens in the large-scale vision transformers, especially for image classification tasks. Typically, they select a small group of essential tokens according to their relevance with the [\texttt{class}] token, then fine-tune the weights of the vision transformer. Such fine-tuning is less practical for dense prediction due to the much heavier computation and GPU memory cost than image classification.In this paper, we focus on a more challenging problem, \ie, accelerating large-scale vision transformers for dense prediction without any additional re-training or fine-tuning. In response to the fact that high-resolution representations are necessary for dense prediction, we present two non-parametric operators, a \emph{token clustering layer} to decrease the number of tokens and a \emph{token reconstruction layer} to increase the number of tokens. The following steps are performed to achieve this: (i) we use the token clustering layer to cluster the neighboring tokens together, resulting in low-resolution representations that maintain the spatial structures; (ii) we apply the following transformer layers only to these low-resolution representations or clustered tokens; and (iii) we use the token reconstruction layer to re-create the high-resolution representations from the refined low-resolution representations. The results obtained by our method are promising on five dense prediction tasks including object detection, semantic segmentation, panoptic segmentation, instance segmentation, and depth estimation. Accordingly, our method accelerates $40\%\uparrow$ FPS and saves $30\%\downarrow$ GFLOPs of ``Segmenter+ViT-L/$16$'' while maintaining $99.5\%$ of the performance on ADE$20$K without fine-tuning the official weights.
Keywords: Classification · semantic segmentation · transformer · high-resolution · Efficient Architecture · Depth Estimation
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Weicong Liang (Key Laboratory of Machine Perception (MOE) School of Intelligence Science and Technology Peking University)
YUHUI YUAN (Microsoft Research Asia)
Henghui Ding (Swiss Federal Institute of Technology)
Xiao Luo (Peking University)
Weihong Lin (Microsoft)
Ding Jia (Peking University)
Zheng Zhang (MSRA)
Chao Zhang (Peking University)
Han Hu (Microsoft Research Asia)

More from the Same Authors