Timezone: »
Multiple object tracking and segmentation requires detecting, tracking, and segmenting objects belonging to a set of given classes. Most approaches only exploit the temporal dimension to address the association problem, while relying on single frame predictions for the segmentation mask itself. We propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich spatio-temporal information for online multiple object tracking and segmentation. PCAN first distills a space-time memory into a set of prototypes and then employs cross-attention to retrieve rich information from the past frames. To segment each object, PCAN adopts a prototypical appearance module to learn a set of contrastive foreground and background prototypes, which are then propagated over time. Extensive experiments demonstrate that PCAN outperforms current video instance tracking and segmentation competition winners on both Youtube-VIS and BDD100K datasets, and shows efficacy to both one-stage and two-stage segmentation frameworks. Code and video resources are available at http://vis.xyz/pub/pcan.
Author Information
Lei Ke (The Hong Kong University of Science and Technology)
Xia Li (Peking University)
Martin Danelljan (ETH Zurich)
Yu-Wing Tai (Kuaishou Technology)
Chi-Keung Tang (The Hong Kong University of Science and Technology)
Fisher Yu (ETH Zurich)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation »
Dates n/a. Room
More from the Same Authors
-
2022 Workshop: Human in the Loop Learning (HiLL) Workshop at NeurIPS 2022 »
Shanghang Zhang · Hao Dong · Wei Pan · Pradeep Ravikumar · Vittorio Ferrari · Fisher Yu · Xin Wang · Zihan Ding -
2022 Poster: Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation »
Xinhang Liu · Jiaben Chen · Huai Yu · Yu-Wing Tai · Chi-Keung Tang -
2021 Poster: Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation »
Ho Kei Cheng · Yu-Wing Tai · Chi-Keung Tang -
2020 Workshop: Machine Learning for Autonomous Driving »
Rowan McAllister · Xinshuo Weng · Daniel Omeiza · Nick Rhinehart · Fisher Yu · German Ros · Vladlen Koltun -
2020 Poster: GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network »
Prune Truong · Martin Danelljan · Luc V Gool · Radu Timofte -
2020 Poster: DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation »
Alexandre Carlier · Martin Danelljan · Alexandre Alahi · Radu Timofte -
2019 Workshop: Machine Learning for Autonomous Driving »
Rowan McAllister · Nicholas Rhinehart · Fisher Yu · Li Erran Li · Anca Dragan -
2016 Workshop: 3D Deep Learning »
Fisher Yu · Joseph Lim · Matthew D Fisher · Qixing Huang · Jianxiong Xiao