`

Timezone: »

 
Poster
A Tensorized Transformer for Language Modeling
Xindian Ma · Peng Zhang · Shuai Zhang · Nan Duan · Yuexian Hou · Ming Zhou · Dawei Song

Wed Dec 11 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #101
in Applications -- Natural Language Processing »

Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We test and verify the proposed attention method on three language modeling tasks (i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task (i.e., WMT-2016 English-German). Multi-linear attention can not only largely compress the model parameters but also obtain performance improvements, compared with a number of language modeling approaches, such as Transformer, Transformer-XL, and Transformer with tensor train decomposition.

Keywords: Attention Models · Natural Language Processing · Deep Learning · Applications -> Matrix and Tensor Factorization; Applications
[ Paper]  [ Poster]  [ Slides] 

Author Information

Xindian Ma (Tianjin University)
Peng Zhang (Tianjin University)
Shuai Zhang (Tianjin University)
Nan Duan (Microsoft Research Asia)
Yuexian Hou (Tianjin University)
Ming Zhou (Microsoft Research)
Dawei Song (Beijing Institute of Technology)

More from the Same Authors