Timezone: »

 
Poster
Improving Transformer with an Admixture of Attention Heads
Tan Nguyen · Tam Nguyen · Hai Do · Khai Nguyen · Vishwanath Saragadam · Minh Pham · Khuong Duy Nguyen · Nhat Ho · Stanley Osher

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #126
in Poster Session 2 »

Transformers with multi-head self-attention have achieved remarkable success in sequence modeling and beyond. However, they suffer from high computational and memory complexities for computing the attention matrix at each head. Recently, it has been shown that those attention matrices lie on a low-dimensional manifold and, thus, are redundant. We propose the Transformer with a Finite Admixture of Shared Heads (FiSHformers), a novel class of efficient and flexible transformers that allow the sharing of attention matrices between attention heads. At the core of FiSHformer is a novel finite admixture model of shared heads (FiSH) that samples attention matrices from a set of global attention matrices. The number of global attention matrices is much smaller than the number of local attention matrices generated. FiSHformers directly learn these global attention matrices rather than the local ones as in other transformers, thus significantly improving the computational and memory efficiency of the model. We empirically verify the advantages of the FiSHformer over the baseline transformers in a wide range of practical applications including language modeling, machine translation, and image classification. On the WikiText-103, IWSLT'14 De-En and WMT'14 En-De, FiSHformers use much fewer floating-point operations per second (FLOPs), memory, and parameters compared to the baseline transformers.

Keywords: transformer · attentions · redundant heads · admixture
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Tan Nguyen (University of California, Los Angeles)
Tam Nguyen (FPT Software)
Hai Do (Vietnam National University Hanoi)
Khai Nguyen (University of Texas, Austin)
Vishwanath Saragadam (Rice University)
Minh Pham (University of California, Los Angeles)
Khuong Duy Nguyen (JAIST)
Nhat Ho (University of Texas at Austin)
Stanley Osher (UCLA)

More from the Same Authors