Timezone: »

 
Poster
Masked Autoencoders that Listen
Po-Yao Huang · Hu Xu · Juncheng Li · Alexei Baevski · Michael Auli · Wojciech Galuba · Florian Metze · Christoph Feichtenhofer

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #912
in Poster Session 4 »

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. Our code and models is available at https://github.com/facebookresearch/AudioMAE.

Keywords: Self-supervised learning · audio classification · audio representation learning
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Po-Yao Huang (Facebook)
Hu Xu (University of Illinois at Chicago)
Juncheng Li (Carnegie Mellon University)
Alexei Baevski (Facebook AI Research)
Michael Auli (Meta AI)
Wojciech Galuba (Meta AI)
Florian Metze (Meta)
Christoph Feichtenhofer (Facebook AI Research)

More from the Same Authors