Timezone: »

Sigsoftmax: Reanalysis of the Softmax Bottleneck
Sekitoshi Kanai · Yasuhiro Fujiwara · Yuki Yamanaka · Shuichi Adachi

Tue Dec 04 07:45 AM -- 09:45 AM (PST) @ Room 517 AB #128

Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an output activation function for breaking the softmax bottleneck without additional parameters. We re-analyze the softmax bottleneck from the perspective of the output set of log-softmax and identify the cause of the softmax bottleneck. On the basis of this analysis, we propose sigsoftmax, which is composed of a multiplication of an exponential function and sigmoid function. Sigsoftmax can break the softmax bottleneck. The experiments on language modeling demonstrate that sigsoftmax and mixture of sigsoftmax outperform softmax and mixture of softmax, respectively.

Author Information

Sekitoshi Kanai (NTT, Keio University)
Yasuhiro Fujiwara (NTT Software Innovation Center)
Yuki Yamanaka (NTT Secure Platform Laboratories)
Shuichi Adachi (Keio University)

More from the Same Authors