Timezone: »
While 360° cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make core feature extraction non-trivial. Convolutional neural networks (CNNs) trained on images from perspective cameras yield “flat" filters, yet 360° images cannot be projected to a single plane without significant distortion. A naive solution that repeatedly projects the viewing sphere to all tangent planes is accurate, but much too computationally intensive for real problems. We propose to learn a spherical convolutional network that translates a planar CNN to process 360° imagery directly in its equirectangular projection. Our approach learns to reproduce the flat filter outputs on 360° data, sensitive to the varying distortion effects across the viewing sphere. The key benefits are 1) efficient feature extraction for 360° images and video, and 2) the ability to leverage powerful pre-trained networks researchers have carefully honed (together with massive labeled image training sets) for perspective images. We validate our approach compared to several alternative methods in terms of both raw CNN output accuracy as well as applying a state-of-the-art “flat" object detector to 360° data. Our method yields the most accurate results while saving orders of magnitude in computation versus the existing exact reprojection solution.
Author Information
Yu-Chuan Su (UT Austin)
Kristen Grauman (University of Texas at Austin)
More from the Same Authors
-
2021 Spotlight: Shaping embodied agent behavior with activity-context priors from egocentric video »
Tushar Nagarajan · Kristen Grauman -
2023 Poster: EgoEnv: Human-centric environment representations from egocentric video »
Tushar Nagarajan · Santhosh Kumar Ramakrishnan · Ruta Desai · James Hillis · Kristen Grauman -
2023 Poster: Self-Supervised Visual Acoustic Matching »
Arjun Somayazulu · Changan Chen · Kristen Grauman -
2023 Poster: Video-Mined Task Graphs for Keystep Recognition in Instructional Videos »
Kumar Ashutosh · Santhosh Kumar Ramakrishnan · Triantafyllos Afouras · Kristen Grauman -
2023 Poster: Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment »
Zihui Xue · Kristen Grauman -
2023 Poster: EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding »
Shuhan Tan · Tushar Nagarajan · Kristen Grauman -
2023 Poster: Single-Stage Visual Query Localization in Egocentric Videos »
Hanwen Jiang · Santhosh Kumar Ramakrishnan · Kristen Grauman -
2022 Poster: SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning »
Changan Chen · Carl Schissler · Sanchit Garg · Philip Kobernik · Alexander Clegg · Paul Calamia · Dhruv Batra · Philip Robinson · Kristen Grauman -
2022 Poster: Few-Shot Audio-Visual Learning of Environment Acoustics »
Sagnik Majumder · Changan Chen · Ziad Al-Halah · Kristen Grauman -
2021 Poster: Shaping embodied agent behavior with activity-context priors from egocentric video »
Tushar Nagarajan · Kristen Grauman -
2020 : Panel Discussion & Closing »
Yejin Choi · Alexei Efros · Chelsea Finn · Kristen Grauman · Quoc V Le · Yann LeCun · Ruslan Salakhutdinov · Eric Xing -
2020 : Q & A and Panel Session with Dan Weld, Kristen Grauman, Scott Yih, Emma Brunskill, and Alex Ratner »
Kristen Grauman · Wen-tau Yih · Alexander Ratner · Emma Brunskill · Douwe Kiela · Daniel S. Weld -
2020 : QA: Kristen Grauman »
Kristen Grauman -
2020 : Invited Talk: Kristen Grauman »
Kristen Grauman -
2020 Poster: Learning Affordance Landscapes for Interaction Exploration in 3D Environments »
Tushar Nagarajan · Kristen Grauman -
2020 Spotlight: Learning Affordance Landscapes for Interaction Exploration in 3D Environments »
Tushar Nagarajan · Kristen Grauman -
2014 Poster: Diverse Sequential Subset Selection for Supervised Video Summarization »
Boqing Gong · Wei-Lun Chao · Kristen Grauman · Fei Sha -
2014 Poster: Predicting Useful Neighborhoods for Lazy Local Learning »
Aron Yu · Kristen Grauman -
2014 Poster: Zero-shot recognition with unreliable attributes »
Dinesh Jayaraman · Kristen Grauman -
2013 Poster: Reshaping Visual Datasets for Domain Adaptation »
Boqing Gong · Kristen Grauman · Fei Sha -
2012 Poster: Semantic Kernel Forests from Multiple Taxonomies »
Sung Ju Hwang · Kristen Grauman · Fei Sha -
2011 Poster: Learning a Tree of Metrics with Disjoint Visual Features »
Sung Ju Hwang · Kristen Grauman · Fei Sha -
2010 Poster: Hashing Hyperplane Queries to Near Points with Applications to Large-Scale Active Learning »
Prateek Jain · Sudheendra Vijayanarasimhan · Kristen Grauman -
2008 Oral: Multi-Level Active Prediction of Useful Image Annotations for Recognition »
Sudheendra N Vijayanarasimhan · Kristen Grauman -
2008 Poster: Multi-Level Active Prediction of Useful Image Annotations for Recognition »
Sudheendra N Vijayanarasimhan · Kristen Grauman -
2008 Poster: Online Metric Learning and Fast Similarity Search »
Prateek Jain · Brian Kulis · Inderjit Dhillon · Kristen Grauman -
2008 Oral: Online Metric Learning and Fast Similarity Search »
Prateek Jain · Brian Kulis · Inderjit Dhillon · Kristen Grauman -
2006 Poster: Approximate Correspondences in High Dimensions »
Kristen Grauman · Trevor Darrell -
2006 Spotlight: Approximate Correspondences in High Dimensions »
Kristen Grauman · Trevor Darrell