Timezone: »

 
Poster
BiT: Robustly Binarized Multi-distilled Transformer
Zechun Liu · Barlas Oguz · Aasish Pappu · Lin Xiao · Scott Yih · Meng Li · Raghuraman Krishnamoorthi · Yashar Mehdad

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #105
in Poster Session 6 »

Modern pre-trained transformers have rapidly advanced the state-of-the-art in machine learning, but have also grown in parameters and computational complexity, making them increasingly difficult to deploy in resource-constrained environments. Binarization of the weights and activations of the network can significantly alleviate these issues, however, is technically challenging from an optimization perspective. In this work, we identify a series of improvements that enables binary transformers at a much higher accuracy than what was possible previously. These include a two-set binarization scheme, a novel elastic binary activation function with learned parameters, and a method to quantize a network to its limit by successively distilling higher precision models into lower precision students. These approaches allow for the first time, fully binarized transformer models that are at a practical level of accuracy, approaching a full-precision BERT baseline on the GLUE language understanding benchmark within as little as 5.9%. Code and models are available at:https://github.com/facebookresearch/bit.

Keywords: Natural Language Processing · transformers · compression · BERT · Binary neural networks
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Zechun Liu (CMU)
Barlas Oguz (Facebook AI)
Aasish Pappu (Meta AI)
Lin Xiao (Meta AI Research)
Scott Yih (Meta AI - FAIR)
Meng Li (Peking University)
Raghuraman Krishnamoorthi (Facebook)
Yashar Mehdad (Facebook)

More from the Same Authors