Timezone: »
Poster
Volumetric Correspondence Networks for Optical Flow
Gengshan Yang · Deva Ramanan
Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #74
Many classic tasks in vision -- such as the estimation of optical flow or stereo disparities -- can be cast as dense correspondence matching. Well-known techniques for doing so make use of a cost volume, typically a 4D tensor of match costs between all pixels in a 2D image and their potential matches in a 2D search window. State-of-the-art (SOTA) deep networks for flow/stereo make use of such volumetric representations as internal layers. However, such layers require significant amounts of memory and compute, making them cumbersome to use in practice. As a result, SOTA networks also employ various heuristics designed to limit volumetric processing, leading to limited accuracy and overfitting. Instead, we introduce several simple modifications that dramatically simplify the use of volumetric layers - (1) volumetric encoder-decoder architectures that efficiently capture large receptive fields, (2) multi-channel cost volumes that capture multi-dimensional notions of pixel similarities, and finally, (3) separable volumetric filtering that significantly reduces computation and parameters while preserving accuracy. Our innovations dramatically improve accuracy over SOTA on standard benchmarks while being significantly easier to work with - training converges in 7X fewer iterations, and most importantly, our networks generalize across correspondence tasks. On-the-fly adaptation of search windows allows us to repurpose optical flow networks for stereo (and vice versa), and can also be used to implement adaptive networks that increase search window sizes on-demand.
Author Information
Gengshan Yang (Carnegie Mellon University)
Deva Ramanan (Carnegie Mellon University)
More from the Same Authors
-
2021 Spotlight: ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction »
Gengshan Yang · Deqing Sun · Varun Jampani · Daniel Vlasic · Forrester Cole · Ce Liu · Deva Ramanan -
2021 : Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting »
Benjamin Wilson · William Qi · Tanmay Agarwal · John Lambert · Jagjeet Singh · Siddhesh Khandelwal · Bowen Pan · Ratnesh Kumar · Andrew Hartnett · Jhony Kaesemodel Pontes · Deva Ramanan · Peter Carr · James Hays -
2021 : The CLEAR Benchmark: Continual LEArning on Real-World Imagery »
Zhiqiu Lin · Jia Shi · Deepak Pathak · Deva Ramanan -
2022 Poster: Continual Learning with Evolving Class Ontologies »
Zhiqiu Lin · Deepak Pathak · Yu-Xiong Wang · Deva Ramanan · Shu Kong -
2022 Poster: Learning to Discover and Detect Objects »
Vladimir Fomenko · Ismail Elezi · Deva Ramanan · Laura Leal-TaixĂ© · Aljosa Osep -
2021 Poster: ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction »
Gengshan Yang · Deqing Sun · Varun Jampani · Daniel Vlasic · Forrester Cole · Ce Liu · Deva Ramanan -
2021 Poster: NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild »
Jason Zhang · Gengshan Yang · Shubham Tulsiani · Deva Ramanan -
2017 Poster: Learning to Model the Tail »
Yu-Xiong Wang · Deva Ramanan · Martial Hebert -
2017 Poster: Attentional Pooling for Action Recognition »
Rohit Girdhar · Deva Ramanan