NeurIPS 2023

Timezone: »

Poster

Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling

Haotao Wang · Ziyu Jiang · Yuning You · Yan Han · Gaowen Liu · Jayanth Srinivasa · Ramana Kompella · Zhangyang "Atlas" Wang

Wed Dec 13 08:45 AM -- 10:45 AM (PST) @ Great Hall & Hall B1+B2 #2016

in Poster Session 3 »

Graph neural networks (GNNs) have found extensive applications in learning from graph data. However, real-world graphs often possess diverse structures and comprise nodes and edges of varying types. To bolster the generalization capacity of GNNs, it has become customary to augment training graph structures through techniques like graph augmentations and large-scale pre-training on a wider array of graphs. Balancing this diversity while avoiding increased computational costs and the notorious trainability issues of GNNs is crucial. This study introduces the concept of Mixture-of-Experts (MoE) to GNNs, with the aim of augmenting their capacity to adapt to a diverse range of training graph structures, without incurring explosive computational overhead. The proposed Graph Mixture of Experts (GMoE) model empowers individual nodes in the graph to dynamically and adaptively select more general information aggregation experts. These experts are trained to capture distinct subgroups of graph structures and to incorporate information with varying hop sizes, where those with larger hop sizes specialize in gathering information over longer distances. The effectiveness of GMoE is validated through a series of experiments on a diverse set of tasks, including graph, node, and link prediction, using the OGB benchmark. Notably, it enhances ROC-AUC by $1.81\%$ in ogbg-molhiv and by $1.40\%$ in ogbg-molbbbp, when compared to the non-MoE baselines. Our code is publicly available at https://github.com/VITA-Group/Graph-Mixture-of-Experts.

[ OpenReview]

Author Information

Haotao Wang (University of Texas at Austin)

Ziyu Jiang (Texas A&M University)

Yuning You (Texas A&M University)

Yan Han (The University of Texas at Austin)

Gaowen Liu (Cisco Research)

Jayanth Srinivasa (Cisco)

Ramana Kompella (Cisco Systems)

Dr. Kompella is the Head of Cisco Research where he leads a team of researchers conduct cutting edge research on topics of emerging interest to Cisco, such as AI/ML, Computer Vision, and Quantum Computing. Prior to this role, he was a co-founder at AppFormix, and also a tenured Associate Professor at Purdue University.