Timezone: »

 
Poster
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Shoufa Chen · Chongjian GE · Zhan Tong · Jiangliu Wang · Yibing Song · Jue Wang · Ping Luo

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #142
in Poster Session 3 »

Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition. A following scenario is to adapt a ViT to various image and video recognition tasks. The adaptation is challenging because of heavy computation and memory storage. Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains.To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently.It possesses several benefits more appealing than prior arts.Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks.Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks.Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains. For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something~v2 and HMDB51, respectively. Code is available at https://github.com/ShoufaChen/AdaptFormer.

Keywords: Efficient Finetuning · Visual Adapter
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Shoufa Chen (The University of Hong Kong)
Chongjian GE (The University of Hong Kong)
Zhan Tong (Tencent AI Lab)
Jiangliu Wang (The Chinese University of Hong Kong)
Yibing Song (Tencent AI Lab)
Jue Wang (Tencent AI Lab)
Ping Luo (The University of Hong Kong)

More from the Same Authors