Timezone: »

 
Poster
IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
Bowen Pan · Rameswar Panda · Yifan Jiang · Zhangyang Wang · Rogerio Feris · Aude Oliva

Thu Dec 09 12:30 AM -- 02:00 AM (PST) @
in Poster Session 5 »
The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. In spite of the impressive success made by transformers in a variety of vision tasks, it still suffers from heavy computation and intensive memory costs. To address this limitation, this paper presents an Interpretability-Aware REDundancy REDuction framework (IA-RED$^2$). We start by observing a large amount of redundant computation, mainly spent on uncorrelated input patches, and then introduce an interpretable module to dynamically and gracefully drop these redundant patches. This novel framework is then extended to a hierarchical structure, where uncorrelated tokens at different stages are gradually removed, resulting in a considerable shrinkage of computational cost. We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4x speed-up for state-of-the-art models like DeiT and TimeSformer, by only sacrificing less than 0.7% accuracy. More importantly, contrary to other acceleration approaches, our method is inherently interpretable with substantial visual evidence, making vision transformer closer to a more human-understandable architecture while being lighter. We demonstrate that the interpretability that naturally emerged in our framework can outperform the raw attention learned by the original visual transformer, as well as those generated by off-the-shelf interpretation methods, with both qualitative and quantitative results. Project Page: http://people.csail.mit.edu/bpan/ia-red/.
Keywords: Deep Learning · Transformers · Vision · Interpretability

Author Information

Bowen Pan (Massachusetts Institute of Technology)
Rameswar Panda (MIT-IBM Watson AI Lab)
Yifan Jiang (The University of Texas at Austin)
Zhangyang Wang (UT Austin)
Rogerio Feris (MIT-IBM Watson AI Lab, IBM Research)
Aude Oliva (Massachusetts Institute of Technology)

More from the Same Authors