Timezone: »
Humans can easily detect and segment moving objects simply by observing how they move, even without knowledge of object semantics. Inspired by this, we develop a zero-shot unsupervised approach for learning object segmentations. The model comprises two visual pathways: an appearance pathway that segments individual RGB images into coherent object regions, and a motion pathway that predicts the flow vector for each region between consecutive video frames. The two pathways jointly reconstruct a new representation called segment flow. This decoupled representation of appearance and motion is trained in a self-supervised manner to reconstruct one frame from another.When pretrained on an unlabeled video corpus, the model can be useful for a variety of applications, including 1) primary object segmentation from a single image in a zero-shot fashion; 2) moving object segmentation from a video with unsupervised test-time adaptation; 3) image semantic segmentation by supervised fine-tuning on a labeled image dataset. We demonstrate encouraging experimental results on all of these tasks using pretrained models.
Author Information
Runtao Liu (Johns Hopkins University)
Zhirong Wu (Microsoft)
Stella Yu (UC Berkeley / ICSI)
Stephen Lin (Microsoft Research)
More from the Same Authors
-
2020 : Paper 62: Instance-wise Depth and Motion Learning from Monocular Videos »
Seokju Lee · Sunghoon Im · Stephen Lin · In So Kweon -
2021 Spotlight: Aligning Pretraining for Detection via Object-Level Contrastive Learning »
Fangyun Wei · Yue Gao · Zhirong Wu · Han Hu · Stephen Lin -
2021 Spotlight: Bootstrap Your Object Detector via Mixed Training »
Mengde Xu · Zheng Zhang · Fangyun Wei · Yutong Lin · Yue Cao · Stephen Lin · Han Hu · Xiang Bai -
2022 Poster: Could Giant Pre-trained Image Models Extract Universal Representations? »
Yutong Lin · Ze Liu · Zheng Zhang · Han Hu · Nanning Zheng · Stephen Lin · Yue Cao -
2022 : Modeling Semantic Correlation and Hierarchy for Real-world Wildlife Recognition »
Dong-Jin Kim · Zhongqi Miao · Yunhui Guo · Stella Yu · Kyle Landolt · Mark Koneff · Travis Harrison -
2022 : Contextual Visual Feature Learning for Zero-Shot Recognition of Human-Object Interactions »
Tsung-Wei Ke · Dong-Jin Kim · Stella Yu · Liang Gou · Liu Ren -
2022 : Multi-band Image Classification with Ultra-Lean Complex-Valued Models »
Utkarsh Singhal · Stella Yu · Zackery Steck · Scott Kangas -
2023 : Associative Transformer Is A Sparse Representation Learner »
Yuwei Sun · Hideya Ochiai · Zhirong Wu · Stephen Lin · Ryota Kanai -
2023 : Associative Transformer Is A Sparse Representation Learner »
Yuwei Sun · Hideya Ochiai · Zhirong Wu · Stephen Lin · Ryota Kanai -
2022 Spotlight: Lightning Talks 2A-3 »
David Buterez · Chengan He · Xuan Kan · Yutong Lin · Konstantin Schürholt · Yu Yang · Louis Annabi · Wei Dai · Xiaotian Cheng · Alexandre Pitti · Ze Liu · Jon Paul Janet · Jun Saito · Boris Knyazev · Mathias Quoy · Zheng Zhang · James Zachary · Steven J Kiddle · Xavier Giro-i-Nieto · Chang Liu · Hejie Cui · Zilong Zhang · Hakan Bilen · Damian Borth · Dino Oglic · Holly Rushmeier · Han Hu · Xiangyang Ji · Yi Zhou · Nanning Zheng · Ying Guo · Pietro Liò · Stephen Lin · Carl Yang · Yue Cao -
2022 Spotlight: Could Giant Pre-trained Image Models Extract Universal Representations? »
Yutong Lin · Ze Liu · Zheng Zhang · Han Hu · Nanning Zheng · Stephen Lin · Yue Cao -
2021 Poster: Aligning Pretraining for Detection via Object-Level Contrastive Learning »
Fangyun Wei · Yue Gao · Zhirong Wu · Han Hu · Stephen Lin -
2021 Poster: Bootstrap Your Object Detector via Mixed Training »
Mengde Xu · Zheng Zhang · Fangyun Wei · Yutong Lin · Yue Cao · Stephen Lin · Han Hu · Xiang Bai -
2020 Poster: RepPoints v2: Verification Meets Regression for Object Detection »
Yihong Chen · Zheng Zhang · Yue Cao · Liwei Wang · Stephen Lin · Han Hu -
2019 : Poster Session »
Jonathan Scarlett · Piotr Indyk · Ali Vakilian · Adrian Weller · Partha P Mitra · Benjamin Aubin · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová · Kristina Monakhova · Joshua Yurtsever · Laura Waller · Hendrik Sommerhoff · Michael Moeller · Rushil Anirudh · Shuang Qiu · Xiaohan Wei · Zhuoran Yang · Jayaraman Thiagarajan · Salman Asif · Michael Gillhofer · Johannes Brandstetter · Sepp Hochreiter · Felix Petersen · Dhruv Patel · Assad Oberai · Akshay Kamath · Sushrut Karmalkar · Eric Price · Ali Ahmed · Zahra Kadkhodaie · Sreyas Mohan · Eero Simoncelli · Carlos Fernandez-Granda · Oscar Leong · Wesam Sakla · Rebecca Willett · Stephan Hoyer · Jascha Sohl-Dickstein · Sam Greydanus · Gauri Jagatap · Chinmay Hegde · Michael Kellman · Jonathan Tamir · Nouamane Laanait · Ousmane Dia · Mirco Ravanelli · Jonathan Binas · Negar Rostamzadeh · Shirin Jalali · Tiantian Fang · Alex Schwing · Sébastien Lachapelle · Philippe Brouillard · Tristan Deleu · Simon Lacoste-Julien · Stella Yu · Arya Mazumdar · Ankit Singh Rawat · Yue Zhao · Jianshu Chen · Xiaoyang Li · Hubert Ramsauer · Gabrio Rizzuti · Nikolaos Mitsakos · Dingzhou Cao · Thomas Strohmer · Yang Li · Pei Peng · Gregory Ongie -
2018 Poster: Recurrent Transformer Networks for Semantic Correspondence »
Seungryong Kim · Stephen Lin · Sangryul Jeon · Dongbo Min · Kwanghoon Sohn -
2018 Spotlight: Recurrent Transformer Networks for Semantic Correspondence »
Seungryong Kim · Stephen Lin · Sangryul Jeon · Dongbo Min · Kwanghoon Sohn