Timezone: »

SVD-Softmax: Fast Softmax Approximation on Large Vocabulary Neural Networks
Kyuhong Shim · Minjae Lee · Iksoo Choi · Yoonho Boo · Wonyong Sung

Mon Dec 04 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #99

We propose a fast approximation method of a softmax function with a very large vocabulary using singular value decomposition (SVD). SVD-softmax targets fast and accurate probability estimation of the topmost probable words during inference of neural network language models. The proposed method transforms the weight matrix used in the calculation of the output vector by using SVD. The approximate probability of each word can be estimated with only a small part of the weight matrix by using a few large singular values and the corresponding elements for most of the words. We applied the technique to language modeling and neural machine translation and present a guideline for good approximation. The algorithm requires only approximately 20\% of arithmetic operations for an 800K vocabulary case and shows more than a three-fold speedup on a GPU.

Author Information

Kyuhong Shim (Seoul National University)
Minjae Lee (Seoul National University)
Iksoo Choi (Seoul National University)
Yoonho Boo (Seoul National University)
Wonyong Sung (Seoul National University)

More from the Same Authors