Timezone: »
Poster
DropCov: A Simple yet Effective Method for Improving Deep Architectures
Qilong Wang · Mingze Gao · Zhaolin Zhang · Jiangtao Xie · Peihua Li · Qinghua Hu
@
Previous works show global covariance pooling (GCP) has great potential to improve deep architectures especially on visual recognition tasks, where post-normalization of GCP plays a very important role in final performance. Although several post-normalization strategies have been studied, these methods pay more close attention to effect of normalization on covariance representations rather than the whole GCP networks, and their effectiveness requires further understanding. Meanwhile, existing effective post-normalization strategies (e.g., matrix power normalization) usually suffer from high computational complexity (e.g., $O(d^{3})$ for $d$-dimensional inputs). To handle above issues, this work first analyzes the effect of post-normalization from the perspective of training GCP networks. Particularly, we for the first time show that \textit{effective post-normalization can make a good trade-off between representation decorrelation and information preservation for GCP, which are crucial to alleviate over-fitting and increase representation ability of deep GCP networks, respectively}. Based on this finding, we can improve existing post-normalization methods with some small modifications, providing further support to our observation. Furthermore, this finding encourages us to propose a novel pre-normalization method for GCP (namely DropCov), which develops an adaptive channel dropout on features right before GCP, aiming to reach trade-off between representation decorrelation and information preservation in a more efficient way. Our DropCov only has a linear complexity of $O(d)$, while being free for inference. Extensive experiments on various benchmarks (i.e., ImageNet-1K, ImageNet-C, ImageNet-A, Stylized-ImageNet, and iNat2017) show our DropCov is superior to the counterparts in terms of efficiency and effectiveness, and provides a simple yet effective method to improve performance of deep architectures involving both deep convolutional neural networks (CNNs) and vision transformers (ViTs).
Author Information
Qilong Wang (Tianjin University)
Mingze Gao (Tianjin University)
Zhaolin Zhang (Tianjin University)
Jiangtao Xie (Dalian University of Technology)
Peihua Li (Dalian University of Technology)
Qinghua Hu (Tianjin University)
More from the Same Authors
-
2023 Poster: Public Opinion Field Effect Fusion in Representation Learning for Trending Topics Diffusion »
Junliang Li · Yang Yajun · Qinghua Hu · Xin Wang · Hong Gao -
2023 Poster: Fairness-guided Few-shot Prompting for Large Language Models »
Huan Ma · Changqing Zhang · Yatao Bian · Lemao Liu · Zhirui Zhang · Peilin Zhao · Shu Zhang · Huazhu Fu · Qinghua Hu · Bingzhe Wu -
2022 Spotlight: DropCov: A Simple yet Effective Method for Improving Deep Architectures »
Qilong Wang · Mingze Gao · Zhaolin Zhang · Jiangtao Xie · Peihua Li · Qinghua Hu -
2022 Spotlight: Lightning Talks 6A-1 »
Ziyi Wang · Nian Liu · Yaming Yang · Qilong Wang · Yuanxin Liu · Zongxin Yang · Yizhao Gao · Yanchen Deng · Dongze Lian · Nanyi Fei · Ziyu Guan · Xiao Wang · Shufeng Kong · Xumin Yu · Daquan Zhou · Yi Yang · Fandong Meng · Mingze Gao · Caihua Liu · Yongming Rao · Zheng Lin · Haoyu Lu · Zhe Wang · Jiashi Feng · Zhaolin Zhang · Deyu Bo · Xinchao Wang · Chuan Shi · Jiangnan Li · Jiangtao Xie · Jie Zhou · Zhiwu Lu · Wei Zhao · Bo An · Jiwen Lu · Peihua Li · Jian Pei · Hao Jiang · Cai Xu · Peng Fu · Qinghua Hu · Yijie Li · Weigang Lu · Yanan Cao · Jianbin Huang · Weiping Wang · Zhao Cao · Jie Zhou -
2021 Poster: Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions »
Huan Ma · Zongbo Han · Changqing Zhang · Huazhu Fu · Joey Tianyi Zhou · Qinghua Hu -
2021 Poster: Temporal-attentive Covariance Pooling Networks for Video Recognition »
Zilin Gao · Qilong Wang · Bingbing Zhang · Qinghua Hu · Peihua Li -
2019 Poster: CPM-Nets: Cross Partial Multi-View Networks »
Changqing Zhang · Zongbo Han · yajie cui · Huazhu Fu · Joey Tianyi Zhou · Qinghua Hu -
2019 Spotlight: CPM-Nets: Cross Partial Multi-View Networks »
Changqing Zhang · Zongbo Han · yajie cui · Huazhu Fu · Joey Tianyi Zhou · Qinghua Hu -
2018 Poster: Global Gated Mixture of Second-order Pooling for Improving Deep Convolutional Neural Networks »
Qilong Wang · Zilin Gao · Jiangtao Xie · Wangmeng Zuo · Peihua Li