Timezone: »

 
Poster
Representing Long-Range Context for Graph Neural Networks with Global Attention
Zhanghao Wu · Paras Jain · Matthew Wright · Azalia Mirhoseini · Joseph Gonzalez · Ion Stoica

Wed Dec 08 04:30 PM -- 06:00 PM (PST) @
in Poster Session 4 »

Graph neural networks are powerful architectures for structured datasets. However, current methods struggle to represent long-range dependencies. Scaling the depth or width of GNNs is insufficient to broaden receptive fields as larger GNNs encounter optimization instabilities such as vanishing gradients and representation oversmoothing, while pooling-based approaches have yet to become as universally useful as in computer vision. In this work, we propose the use of Transformer-based self-attention to learn long-range pairwise relationships, with a novel “readout” mechanism to obtain a global graph embedding. Inspired by recent computer vision results that find position-invariant attention performant in learning long-range relationships, our method, which we call GraphTrans, applies a permutation-invariant Transformer module after a standard GNN module. This simple architecture leads to state-of-the-art results on several graph classification tasks, outperforming methods that explicitly encode graph structure. Our results suggest that purely-learning-based approaches without graph structure may be suitable for learning high-level, long-range relationships on graphs. Code for GraphTrans is available at https://github.com/ucbrise/graphtrans.

Keywords: Deep Learning · Optimization · Machine Learning · Transformers · Vision · Graph Learning

Author Information

Zhanghao Wu (University of California Berkeley)
Paras Jain (University of California Berkeley)
Matthew Wright (University of California Berkeley)
Azalia Mirhoseini (Google Brain)
Joseph Gonzalez (UC Berkeley)
Ion Stoica (UC Berkeley)

More from the Same Authors