Timezone: »
For video recognition task, a global representation summarizing the whole contents of the video snippets plays an important role for the final performance. However, existing video architectures usually generate it by using a simple, global average pooling (GAP) method, which has limited ability to capture complex dynamics of videos. For image recognition task, there exist evidences showing that covariance pooling has stronger representation ability than GAP. Unfortunately, such plain covariance pooling used in image recognition is an orderless representative, which cannot model spatio-temporal structure inherent in videos. Therefore, this paper proposes a Temporal-attentive Covariance Pooling (TCP), inserted at the end of deep architectures, to produce powerful video representations. Specifically, our TCP first develops a temporal attention module to adaptively calibrate spatio-temporal features for the succeeding covariance pooling, approximatively producing attentive covariance representations. Then, a temporal covariance pooling performs temporal pooling of the attentive covariance representations to characterize both intra-frame correlations and inter-frame cross-correlations of the calibrated features. As such, the proposed TCP can capture complex temporal dynamics. Finally, a fast matrix power normalization is introduced to exploit geometry of covariance representations. Note that our TCP is model-agnostic and can be flexibly integrated into any video architectures, resulting in TCPNet for effective video recognition. The extensive experiments on six benchmarks (e.g., Kinetics, Something-Something V1 and Charades) using various video architectures show our TCPNet is clearly superior to its counterparts, while having strong generalization ability. The source code is publicly available.
Author Information
Zilin Gao (Dalian University of Technology)
Qilong Wang (Tianjin University)
Bingbing Zhang (Dalian University of Technology)
Qinghua Hu (Tianjin University)
Peihua Li (Dalian University of Technology)
More from the Same Authors
-
2022 Poster: DropCov: A Simple yet Effective Method for Improving Deep Architectures »
Qilong Wang · Mingze Gao · Zhaolin Zhang · Jiangtao Xie · Peihua Li · Qinghua Hu -
2022 Spotlight: DropCov: A Simple yet Effective Method for Improving Deep Architectures »
Qilong Wang · Mingze Gao · Zhaolin Zhang · Jiangtao Xie · Peihua Li · Qinghua Hu -
2022 Spotlight: Lightning Talks 6A-1 »
Ziyi Wang · Nian Liu · Yaming Yang · Qilong Wang · Yuanxin Liu · Zongxin Yang · Yizhao Gao · Yanchen Deng · Dongze Lian · Nanyi Fei · Ziyu Guan · Xiao Wang · Shufeng Kong · Xumin Yu · Daquan Zhou · Yi Yang · Fandong Meng · Mingze Gao · Caihua Liu · Yongming Rao · Zheng Lin · Haoyu Lu · Zhe Wang · Jiashi Feng · Zhaolin Zhang · Deyu Bo · Xinchao Wang · Chuan Shi · Jiangnan Li · Jiangtao Xie · Jie Zhou · Zhiwu Lu · Wei Zhao · Bo An · Jiwen Lu · Peihua Li · Jian Pei · Hao Jiang · Cai Xu · Peng Fu · Qinghua Hu · Yijie Li · Weigang Lu · Yanan Cao · Jianbin Huang · Weiping Wang · Zhao Cao · Jie Zhou -
2021 Poster: Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions »
Huan Ma · Zongbo Han · Changqing Zhang · Huazhu Fu · Joey Tianyi Zhou · Qinghua Hu -
2019 Poster: CPM-Nets: Cross Partial Multi-View Networks »
Changqing Zhang · Zongbo Han · yajie cui · Huazhu Fu · Joey Tianyi Zhou · Qinghua Hu -
2019 Spotlight: CPM-Nets: Cross Partial Multi-View Networks »
Changqing Zhang · Zongbo Han · yajie cui · Huazhu Fu · Joey Tianyi Zhou · Qinghua Hu -
2018 Poster: Global Gated Mixture of Second-order Pooling for Improving Deep Convolutional Neural Networks »
Qilong Wang · Zilin Gao · Jiangtao Xie · Wangmeng Zuo · Peihua Li