Timezone: »
Transformer is a transformative framework for deep learning which models sequential data and has achieved remarkable performance on a wide range of tasks, but with high computational and energy cost. To improve its efficiency, a popular choice is to compress the models via binarization which constrains the floating-point values into binary ones to save resource consumption owing to cheap bitwise operations significantly. However, existing binarization methods only aim at minimizing the information loss for the input distribution statistically, while ignoring the pairwise similarity modeling at the core of the attention mechanism. To this end, we propose a new binarization paradigm customized to high-dimensional softmax attention via kernelized hashing, called EcoFormer, to map the original queries and keys into low-dimensional binary codes in Hamming space. The kernelized hash functions are learned to match the ground-truth similarity relations extracted from the attention map in a self-supervised way. Based on the equivalence between the inner product of binary codes and the Hamming distance as well as the associative property of matrix multiplication, we can approximate the attention in linear complexity by expressing it as a dot-product of binary codes. Moreover, the compact binary representations of queries and keys in EcoFormer enable us to replace most of the expensive multiply-accumulate operations in attention with simple accumulations to save considerable on-chip energy footprint on edge devices. Extensive experiments on both vision and language tasks show that EcoFormer consistently achieves comparable performance with standard attentions while consuming much fewer resources. For example, based on PVTv2-B0 and ImageNet-1K, EcoFormer achieves a 73% reduction in on-chip energy footprint with only a slight performance drop of 0.33% compared to the standard attention. Code is available at https://github.com/ziplab/EcoFormer.
Author Information
Jing Liu (Monash University)
Zizheng Pan (Monash University)
Haoyu He (Monash University)
Jianfei Cai (Monash University)
Bohan Zhuang (Monash University)
Dr. Bohan Zhuang is a tenure-track assistant professor at the Faculty of Information Technology, Monash University, Australia. Previously, he was a Senior Research Fellow and finished his PhD in Computer Science at the University of Adelaide, where he was advised by Prof. Ian Reid and Prof. Chunhua Shen. During his undergraduate study, he was fortunately supervised by Prof. Huchuan Lu. His main research topic is efficient deep learning computing. He has also been contributing to a wide span of applications in Machine Learning and Computer Vision. Apart from academic, he is a music enthusiast and have been playing piano since 4 years old.
More from the Same Authors
-
2022 Poster: MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation »
Chuanxia Zheng · Tung-Long Vuong · Jianfei Cai · Dinh Phung -
2023 Poster: PTQD: Accurate Post-Training Quantization for Diffusion Models »
Yefei He · Luping Liu · Jing Liu · 威佳 吴 · Bohan Zhuang · Hong Zhou -
2023 Poster: Mask Propagation for Efficient Video Semantic Segmentation »
Yuetian Weng · Mingfei Han · Haoyu He · Mingjie Li · Lina Yao · Xiaojun Chang · Bohan Zhuang -
2022 Spotlight: Lightning Talks 6B-4 »
Junjie Chen · Chuanxia Zheng · JINLONG LI · Yu Shi · Shichao Kan · Yu Wang · Fermín Travi · Ninh Pham · Lei Chai · Guobing Gan · Tung-Long Vuong · Gonzalo Ruarte · Tao Liu · Li Niu · Jingjing Zou · Zequn Jie · Peng Zhang · Ming LI · Yixiong Liang · Guolin Ke · Jianfei Cai · Gaston Bujia · Sunzhu Li · Siyuan Zhou · Jingyang Lin · Xu Wang · Min Li · Zhuoming Chen · Qing Ling · Xiaolin Wei · Xiuqing Lu · Shuxin Zheng · Dinh Phung · Yigang Cen · Jianlou Si · Juan Esteban Kamienkowski · Jianxin Wang · Chen Qian · Lin Ma · Benyou Wang · Yingwei Pan · Tie-Yan Liu · Liqing Zhang · Zhihai He · Ting Yao · Tao Mei -
2022 Spotlight: Lightning Talks 6B-3 »
Lingfeng Yang · Yao Lai · Zizheng Pan · Zhenyu Wang · Weicong Liang · Chuanyang Zheng · Jian-Wei Zhang · Peng Jin · Jing Liu · Xiuying Wei · Yao Mu · Xiang Li · YUHUI YUAN · Zizheng Pan · Yifan Sun · Yunchen Zhang · Jianfei Cai · Hao Luo · zheyang li · Jinfa Huang · Haoyu He · Yi Yang · Ping Luo · Fenglin Liu · Henghui Ding · Borui Zhao · Xiangguo Zhang · Kai Zhang · Pichao WANG · Bohan Zhuang · Wei Chen · Ruihao Gong · Zhi Yang · Xian Wu · Feng Ding · Jianfei Cai · Xiao Luo · Renjie Song · Weihong Lin · Jian Yang · Wenming Tan · Bohan Zhuang · Shanghang Zhang · Shen Ge · Fan Wang · Qi Zhang · Guoli Song · Jun Xiao · Hao Li · Ding Jia · David Clifton · Ye Ren · Fengwei Yu · Zheng Zhang · Jie Chen · Shiliang Pu · Xianglong Liu · Chao Zhang · Han Hu -
2022 Spotlight: EcoFormer: Energy-Saving Attention with Linear Complexity »
Jing Liu · Zizheng Pan · Haoyu He · Jianfei Cai · Bohan Zhuang -
2022 Spotlight: Fast Vision Transformers with HiLo Attention »
Zizheng Pan · Jianfei Cai · Bohan Zhuang -
2022 Spotlight: MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation »
Chuanxia Zheng · Tung-Long Vuong · Jianfei Cai · Dinh Phung -
2022 Poster: Fast Vision Transformers with HiLo Attention »
Zizheng Pan · Jianfei Cai · Bohan Zhuang