Timezone: »

 
Poster
Self-Supervised Generation of Spatial Audio for 360° Video
Pedro Morgado · Nuno Nvasconcelos · Timothy Langlois · Oliver Wang

Thu Dec 06 07:45 AM -- 09:45 AM (PST) @ Room 517 AB #116
in Thu Poster Session A »

We introduce an approach to convert mono audio recorded by a 360° video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360° video viewing, but spatial audio microphones are still rare in current 360° video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis from the audio and 360° video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360° videos uploaded with spatial audio. During training, ground truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach we show that it is possible to infer the spatial localization of sounds based only on a synchronized 360° video and the mono audio track.

Keywords: Applications

Author Information

Pedro Morgado (University of California, San Diego)
Nuno Nvasconcelos (UC San Diego)
Timothy Langlois (Adobe Systems Inc)
Oliver Wang (Adobe Research)

More from the Same Authors