Timezone: »

 
Poster
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
Changan Chen · Carl Schissler · Sanchit Garg · Philip Kobernik · Alexander Clegg · Paul Calamia · Dhruv Batra · Philip Robinson · Kristen Grauman

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #1026
in Poster Session 5 »

We introduce SoundSpaces 2.0, a platform for on-the-fly geometry-based audio rendering for 3D environments. Given a 3D mesh of a real-world environment, SoundSpaces can generate highly realistic acoustics for arbitrary sounds captured from arbitrary microphone locations. Together with existing 3D visual assets, it supports an array of audio-visual research tasks, such as audio-visual navigation, mapping, source localization and separation, and acoustic matching. Compared to existing resources, SoundSpaces 2.0 has the advantages of allowing continuous spatial sampling, generalization to novel environments, and configurable microphone and material properties. To our knowledge, this is the first geometry-based acoustic simulation that offers high fidelity and realism while also being fast enough to use for embodied learning. We showcase the simulator's properties and benchmark its performance against real-world audio measurements. In addition, we demonstrate two downstream tasks---embodied navigation and far-field automatic speech recognition---and highlight sim2real performance for the latter. SoundSpaces 2.0 is publicly available to facilitate wider research for perceptual systems that can both see and hear.

Keywords: Sim2Real · Audio-Visual Learning · acoustic simulation · visual acoustic learning · embodied learning
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Changan Chen (University of Texas, Austin)
Carl Schissler (Meta)
Sanchit Garg
Philip Kobernik (University of California, Santa Barbara)
Alexander Clegg (Facebook (FAIR Labs))
Alexander Clegg

Research Engineer working on the intersection of computer animation, physical simulation, robotics, and ML in the Habitat team at FAIR. Completed a CS PhD at Georgia Tech, 2019.

Paul Calamia (Reality Labs Research at Meta)
Dhruv Batra (FAIR (Meta) / Georgia Tech)
Philip Robinson (Meta)
Kristen Grauman (University of Texas at Austin)

More from the Same Authors