Timezone: »

 
Poster
BERT Loses Patience: Fast and Robust Inference with Early Exit
Wangchunshu Zhou · Canwen Xu · Tao Ge · Julian McAuley · Ke Xu · Furu Wei

Wed Dec 09 09:00 AM -- 11:00 AM (PST) @ Poster Session 3 #981
in Poster Session 4 »

In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM). To achieve this, our approach couples an internal-classifier with each layer of a PLM and dynamically stops inference when the intermediate predictions of the internal classifiers do not change for a pre-defined number of steps. Our approach improves inference efficiency as it allows the model to make a prediction with fewer layers. Meanwhile, experimental results with an ALBERT model show that our method can improve the accuracy and robustness of the model by preventing it from overthinking and exploiting multiple classifiers for prediction, yielding a better accuracy-speed trade-off compared to existing early exit methods.

Author Information

Wangchunshu Zhou (Beihang University)
Canwen Xu (UC San Diego)
Tao Ge (Microsoft Research Asia)
Julian McAuley (UCSD)
Ke Xu (Beihang University)
Furu Wei (Microsoft Research Asia)

More from the Same Authors