Timezone: »
Spatial-wise dynamic convolution has become a promising approach to improving the inference efficiency of deep networks. By allocating more computation to the most informative pixels, such an adaptive inference paradigm reduces the spatial redundancy in image features and saves a considerable amount of unnecessary computation. However, the theoretical efficiency achieved by previous methods can hardly translate into a realistic speedup, especially on the multi-core processors (e.g. GPUs). The key challenge is that the existing literature has only focused on designing algorithms with minimal computation, ignoring the fact that the practical latency can also be influenced by scheduling strategies and hardware properties. To bridge the gap between theoretical computation and practical efficiency, we propose a latency-aware spatial-wise dynamic network (LASNet), which performs coarse-grained spatially adaptive inference under the guidance of a novel latency prediction model. The latency prediction model can efficiently estimate the inference latency of dynamic networks by simultaneously considering algorithms, scheduling strategies, and hardware properties. We use the latency predictor to guide both the algorithm design and the scheduling optimization on various hardware platforms. Experiments on image classification, object detection and instance segmentation demonstrate that the proposed framework significantly improves the practical inference efficiency of deep networks. For example, the average latency of a ResNet-101 on the ImageNet validation set could be reduced by 36% and 46% on a server GPU (Nvidia Tesla-V100) and an edge device (Nvidia Jetson TX2 GPU) respectively without sacrificing the accuracy. Code is available at https://github.com/LeapLabTHU/LASNet.
Author Information
Yizeng Han (Department of Automation, Tsinghua University)
Zhihang Yuan (Peking University, Tsinghua University)
Yifan Pu (Department of Automation, Tsinghua University)
Chenhao Xue (Peking University)
Shiji Song (Department of Automation, Tsinghua University)
Guangyu Sun (Peking University)
Gao Huang (Cornell University)
More from the Same Authors
-
2022 Poster: Contrastive Language-Image Pre-Training with Knowledge Graphs »
Xuran Pan · Tianzhu Ye · Dongchen Han · Shiji Song · Gao Huang -
2022 Poster: Provable General Function Class Representation Learning in Multitask Bandits and MDP »
Rui Lu · Andrew Zhao · Simon Du · Gao Huang -
2022 Poster: A Mixture Of Surprises for Unsupervised Reinforcement Learning »
Andrew Zhao · Matthieu Lin · Yangguang Li · Yong-jin Liu · Gao Huang -
2022 Poster: Efficient Knowledge Distillation from Model Checkpoints »
Chaofei Wang · Qisen Yang · Rui Huang · Shiji Song · Gao Huang -
2022 : Boosting Offline Reinforcement Learning via Data Resampling »
Yang Yue · Bingyi Kang · Xiao Ma · Zhongwen Xu · Gao Huang · Shuicheng Yan -
2023 : Facilitating Battery Swapping Services for Freight Trucks with Spatial-Temporal Demand Prediction »
Linyu Liu · Zhen Dai · Shiji Song · Xiaocheng Li · Guanting Chen -
2023 Poster: Rank-DETR for High Quality Object Detection »
Yifan Pu · Weicong Liang · Yiduo Hao · YUHUI YUAN · Yukang Yang · Chao Zhang · Han Hu · Gao Huang -
2023 Poster: MIM4DD: Mutual Information Maximization for Dataset Distillation »
Yuzhang Shang · Zhihang Yuan · Yan Yan -
2023 Poster: Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL »
Yang Yue · Rui Lu · Bingyi Kang · Shiji Song · Gao Huang -
2023 Poster: Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning »
Shenzhi Wang · Qisen Yang · Jiawei Gao · Matthieu Lin · HAO CHEN · Liwei Wu · Ning Jia · Shiji Song · Gao Huang -
2022 Spotlight: Lightning Talks 4A-4 »
Yunhao Tang · LING LIANG · Thomas Chau · Daeha Kim · Junbiao Cui · Rui Lu · Lei Song · Byung Cheol Song · Andrew Zhao · Remi Munos · Ćukasz Dudziak · Jiye Liang · Ke Xue · Kaidi Xu · Mark Rowland · Hongkai Wen · Xing Hu · Xiaobin Huang · Simon Du · Nicholas Lane · Chao Qian · Lei Deng · Bernardo Avila Pires · Gao Huang · Will Dabney · Mohamed Abdelfattah · Yuan Xie · Marc Bellemare -
2022 Spotlight: Provable General Function Class Representation Learning in Multitask Bandits and MDP »
Rui Lu · Andrew Zhao · Simon Du · Gao Huang -
2022 Spotlight: Lightning Talks 1B-3 »
Chaofei Wang · Qixun Wang · Jing Xu · Long-Kai Huang · Xi Weng · Fei Ye · Harsh Rangwani · shrinivas ramasubramanian · Yifei Wang · Qisen Yang · Xu Luo · Lei Huang · Adrian G. Bors · Ying Wei · Xinglin Pan · Sho Takemori · Hong Zhu · Rui Huang · Lei Zhao · Yisen Wang · Kato Takashi · Shiji Song · Yanan Li · Rao Anwer · Yuhei Umeda · Salman Khan · Gao Huang · Wenjie Pei · Fahad Shahbaz Khan · Venkatesh Babu R · Zenglin Xu -
2022 Spotlight: Efficient Knowledge Distillation from Model Checkpoints »
Chaofei Wang · Qisen Yang · Rui Huang · Shiji Song · Gao Huang -
2021 Poster: Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition »
Yulin Wang · Rui Huang · Shiji Song · Zeyi Huang · Gao Huang -
2020 Poster: Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification »
Yulin Wang · Kangchen Lv · Rui Huang · Shiji Song · Le Yang · Gao Huang -
2019 Poster: Regularized Anderson Acceleration for Off-Policy Deep Reinforcement Learning »
Wenjie Shi · Shiji Song · Hui Wu · Ya-Chu Hsu · Cheng Wu · Gao Huang -
2019 Poster: Implicit Semantic Data Augmentation for Deep Networks »
Yulin Wang · Xuran Pan · Shiji Song · Hong Zhang · Gao Huang · Cheng Wu -
2019 Poster: Generalization in Generative Adversarial Networks: A Novel Perspective from Privacy Protection »
Bingzhe Wu · Shiwan Zhao · Chaochao Chen · Haoyang Xu · Li Wang · Xiaolu Zhang · Guangyu Sun · Jun Zhou -
2016 Poster: Supervised Word Mover's Distance »
Gao Huang · Chuan Guo · Matt J Kusner · Yu Sun · Fei Sha · Kilian Weinberger -
2016 Oral: Supervised Word Mover's Distance »
Gao Huang · Chuan Guo · Matt J Kusner · Yu Sun · Fei Sha · Kilian Weinberger