Skip to yearly menu bar Skip to main content


Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals

Tam Nguyen · Tan Nguyen · Richard Baraniuk

Great Hall & Hall B1+B2 (level 1) #1924
[ ]
Tue 12 Dec 3:15 p.m. PST — 5:15 p.m. PST


Transformers have achieved remarkable success in a wide range of natural language processing and computer vision applications. However, the representation capacity of a deep transformer model is degraded due to the over-smoothing issue in which the token representations become identical when the model's depth grows. In this work, we show that self-attention layers in transformers minimize a functional which promotes smoothness, thereby causing token uniformity. We then propose a novel regularizer that penalizes the norm of the difference between the smooth output tokens from self-attention and the input tokens to preserve the fidelity of the tokens. Minimizing the resulting regularized energy functional, we derive the Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO), a novel class of transformer models that can mitigate the over-smoothing issue. We empirically demonstrate the advantages of NeuTRENO over the baseline transformers and state-of-the-art methods in reducing the over-smoothing of token representations on various practical tasks, including object classification, image segmentation, and language modeling.

Chat is not available.