Timezone: »

 
Poster
Normalization Helps Training of Quantized LSTM
Lu Hou · Jinhua Zhu · James Kwok · Fei Gao · Tao Qin · Tie-Yan Liu

Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #104
in Deep Learning -- Efficient Inference Methods »

The long-short-term memory (LSTM), though powerful, is memory and computation expensive. To alleviate this problem, one approach is to compress its weights by quantization. However, existing quantization methods usually have inferior performance when used on LSTMs. In this paper, we first show theoretically that training a quantized LSTM is difficult because quantization makes the exploding gradient problem more severe, particularly when the LSTM weight matrices are large. We then show that the popularly used weight/layer/batch normalization schemes can help stabilize the gradient magnitude in training quantized LSTMs. Empirical results show that the normalized quantized LSTMs achieve significantly better results than their unnormalized counterparts. Their performance is also comparable with the full-precision LSTM, while being much smaller in size.

Keywords: Recurrent Networks · Efficient Inference Methods · Deep Learning
[ Paper]  [ Poster] 

Author Information

Lu Hou (Huawei Technologies Co., Ltd)
Jinhua Zhu (University of Science and Technology of China)
James Kwok (Hong Kong University of Science and Technology)
Fei Gao (University of Chinese Academy of Sciences)
Tao Qin (Microsoft Research)
Tie-Yan Liu (Microsoft Research)

More from the Same Authors