Timezone: »
Graph neural architecture search (GraphNAS) has recently aroused considerable attention in both academia and industry. However, two key challenges seriously hinder the further research of GraphNAS. First, since there is no consensus for the experimental setting, the empirical results in different research papers are often not comparable and even not reproducible, leading to unfair comparisons. Secondly, GraphNAS often needs extensive computations, which makes it highly inefficient and inaccessible to researchers without access to large-scale computation. To solve these challenges, we propose NAS-Bench-Graph, a tailored benchmark that supports unified, reproducible, and efficient evaluations for GraphNAS. Specifically, we construct a unified, expressive yet compact search space, covering 26,206 unique graph neural network (GNN) architectures and propose a principled evaluation protocol. To avoid unnecessary repetitive training, we have trained and evaluated all of these architectures on nine representative graph datasets, recording detailed metrics including train, validation, and test performance in each epoch, the latency, the number of parameters, etc. Based on our proposed benchmark, the performance of GNN architectures can be directly obtained by a look-up table without any further computation, which enables fair, fully reproducible, and efficient comparisons. To demonstrate its usage, we make in-depth analyses of our proposed NAS-Bench-Graph, revealing several interesting findings for GraphNAS. We also showcase how the benchmark can be easily compatible with GraphNAS open libraries such as AutoGL and NNI. To the best of our knowledge, our work is the first benchmark for graph neural architecture search.
Author Information
Yijian Qin (Tsinghua University, Tsinghua University)
Ziwei Zhang (Tsinghua University)
Xin Wang (Tsinghua University)
Zeyang Zhang (Tsinghua University)
Wenwu Zhu (Tsinghua University)
More from the Same Authors
-
2022 Poster: Module-Aware Optimization for Auxiliary Learning »
Hong Chen · Xin Wang · Yue Liu · Yuwei Zhou · Chaoyu Guan · Wenwu Zhu -
2022 Poster: Learning Invariant Graph Representations for Out-of-Distribution Generalization »
Haoyang Li · Ziwei Zhang · Xin Wang · Wenwu Zhu -
2022 Poster: Dynamic Graph Neural Networks Under Spatio-Temporal Distribution Shift »
Zeyang Zhang · Xin Wang · Ziwei Zhang · Haoyang Li · Zhou Qin · Wenwu Zhu -
2023 Poster: Fused Gromov-Wasserstein Graph Mixup for Graph-level Classifications »
Xinyu Ma · Xu Chu · Yasha Wang · Yang Lin · Junfeng Zhao · Liantao Ma · Wenwu Zhu -
2023 Poster: Environment-Aware Dynamic Graph Learning for Out-of-Distribution Generalization »
Haonan Yuan · Qingyun Sun · Xingcheng Fu · Ziwei Zhang · Cheng Ji · Hao Peng · Jianxin Li -
2023 Poster: Unsupervised Graph Neural Architecture Search with Disentangled Self-Supervision »
Zeyang Zhang · Xin Wang · Ziwei Zhang · Guangyao Shen · Shiqi Shen · Wenwu Zhu -
2023 Poster: Multi-task Graph Neural Architecture Search with Task-aware Collaboration and Curriculum »
Yijian Qin · Xin Wang · Ziwei Zhang · Hong Chen · Wenwu Zhu -
2023 Poster: Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts »
Zeyang Zhang · Xin Wang · Ziwei Zhang · Zhou Qin · Weigao Wen · Hui Xue' · Haoyang Li · Wenwu Zhu -
2023 Poster: Joint Data-Task Generation for Auxiliary Learning »
Hong Chen · Xin Wang · Yuwei Zhou · Yijian Qin · Chaoyu Guan · Wenwu Zhu -
2022 Spotlight: NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search »
Yijian Qin · Ziwei Zhang · Xin Wang · Zeyang Zhang · Wenwu Zhu -
2022 Poster: On the Convergence of Stochastic Multi-Objective Gradient Manipulation and Beyond »
Shiji Zhou · Wenpeng Zhang · Jiyan Jiang · Wenliang Zhong · Jinjie GU · Wenwu Zhu -
2021 Poster: Asynchronous Decentralized Online Learning »
Jiyan Jiang · Wenpeng Zhang · Jinjie GU · Wenwu Zhu -
2021 Poster: Curriculum Disentangled Recommendation with Noisy Multi-feedback »
Hong Chen · Yudong Chen · Xin Wang · Ruobing Xie · Rui Wang · Feng Xia · Wenwu Zhu -
2021 Poster: Disentangled Contrastive Learning on Graphs »
Haoyang Li · Xin Wang · Ziwei Zhang · Zehuan Yuan · Hang Li · Wenwu Zhu -
2021 Poster: Graph Differentiable Architecture Search with Structure Learning »
Yijian Qin · Xin Wang · Zeyang Zhang · Wenwu Zhu -
2021 Poster: Not All Low-Pass Filters are Robust in Graph Convolutional Networks »
Heng Chang · Yu Rong · Tingyang Xu · Yatao Bian · Shiji Zhou · Xin Wang · Junzhou Huang · Wenwu Zhu -
2020 Poster: Implicit Graph Neural Networks »
Fangda Gu · Heng Chang · Wenwu Zhu · Somayeh Sojoudi · Laurent El Ghaoui -
2019 Poster: Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos »
Yitian Yuan · Lin Ma · Jingwen Wang · Wei Liu · Wenwu Zhu -
2019 Poster: Learning Disentangled Representations for Recommendation »
Jianxin Ma · Chang Zhou · Peng Cui · Hongxia Yang · Wenwu Zhu -
2018 Poster: Weakly Supervised Dense Event Captioning in Videos »
Xin Wang · Wenbing Huang · Chuang Gan · Jingdong Wang · Wenwu Zhu · Junzhou Huang