Timezone: »

 
Poster
Transformers from an Optimization Perspective
Yongyi Yang · zengfeng Huang · David P Wipf

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #228
in Poster Session 4 »

Deep learning models such as the Transformer are often constructed by heuristics and experience. To provide a complementary foundation, in this work we study the following problem: Is it possible to find an energy function underlying the Transformer model, such that descent steps along this energy correspond with the Transformer forward pass? By finding such a function, we can reinterpret Transformers as the unfolding of an interpretable optimization process. This unfolding perspective has been frequently adopted in the past to elucidate more straightforward deep models such as MLPs and CNNs; however, it has thus far remained elusive obtaining a similar equivalence for more complex models with self-attention mechanisms like the Transformer. To this end, we first outline several major obstacles before providing companion techniques to at least partially address them, demonstrating for the first time a close association between energy function minimization and deep layers with self-attention. This interpretation contributes to our intuition and understanding of Transformers, while potentially laying the ground-work for new model designs.

Keywords: transformers · self-attention · unfolded optimization · energy function minimization
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Yongyi Yang (University of Michigan)
zengfeng Huang (Fudan University)
David P Wipf (AWS)

More from the Same Authors