Timezone: »
Attentional models are distinctly characterized by their ability to learn relative importance, i.e., assigning a different weight to input values. This paper proposes a new quasi-attention that is compositional in nature, i.e., learning whether to \textit{add}, \textit{subtract} or \textit{nullify} a certain vector when learning representations. This is strongly contrasted with vanilla attention, which simply re-weights input tokens. Our proposed \textit{Compositional De-Attention} (CoDA) is fundamentally built upon the intuition of both similarity and dissimilarity (negative affinity) when computing affinity scores, benefiting from a greater extent of expressiveness. We evaluate CoDA on six NLP tasks, i.e. open domain question answering, retrieval/ranking, natural language inference, machine translation, sentiment analysis and text2code generation. We obtain promising experimental results, achieving state-of-the-art performance on several tasks/datasets.
Author Information
Yi Tay (NTU, Singapore)
Anh Tuan Luu (MIT CSAIL)
Aston Zhang (Amazon AI)
Shuohang Wang (Singapore Management University)
Siu Cheung Hui (Nanyang Technological University)
More from the Same Authors
-
2021 : Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models »
Boxin Wang · Chejian Xu · Shuohang Wang · Zhe Gan · Yu Cheng · Jianfeng Gao · Ahmed Awadallah · Bo Li -
2021 Poster: G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminators »
Yunhui Long · Boxin Wang · Zhuolin Yang · Bhavya Kailkhura · Aston Zhang · Carl Gunter · Bo Li -
2021 Poster: Self-Instantiated Recurrent Units with Dynamic Soft Recursion »
Aston Zhang · Yi Tay · Yikang Shen · Alvin Chan · SHUAI ZHANG -
2019 Poster: Quaternion Knowledge Graph Embeddings »
SHUAI ZHANG · Yi Tay · Lina Yao · Qi Liu -
2018 Poster: Densely Connected Attention Propagation for Reading Comprehension »
Yi Tay · Anh Tuan Luu · Siu Cheung Hui · Jian Su -
2018 Poster: Recurrently Controlled Recurrent Networks »
Yi Tay · Anh Tuan Luu · Siu Cheung Hui