Timezone: »

 
Poster
TNASP: A Transformer-based NAS Predictor with a Self-evolution Framework
Shun Lu · Jixiang Li · Jianchao Tan · Sen Yang · Ji Liu

Wed Dec 08 04:30 PM -- 06:00 PM (PST) @ Virtual
in Poster Session 4 »

Predictor-based Neural Architecture Search (NAS) continues to be an important topic because it aims to mitigate the time-consuming search procedure of traditional NAS methods. A promising performance predictor determines the quality of final searched models in predictor-based NAS methods. Most existing predictor-based methodologies train model-based predictors under a proxy dataset setting, which may suffer from the accuracy decline and the generalization problem, mainly due to their poor abilities to represent spatial topology information of the graph structure data. Besides the poor encoding for spatial topology information, these works did not take advantage of the temporal information such as historical evaluations during training. Thus, we propose a Transformer-based NAS performance predictor, associated with a Laplacian matrix based positional encoding strategy, which better represents topology information and achieves better performance than previous state-of-the-art methods on NAS-Bench-101, NAS-Bench-201, and DARTS search space. Furthermore, we also propose a self-evolution framework that can fully utilize temporal information as guidance. This framework iteratively involves the evaluations of previously predicted results as constraints into current optimization iteration, thus further improving the performance of our predictor. Such framework is model-agnostic, thus can enhance performance on various backbone structures for the prediction task. Our proposed method helped us rank 2nd among all teams in CVPR 2021 NAS Competition Track 2: Performance Prediction Track.

Keywords: Deep Learning · Optimization · Transformers · Graph Learning

Author Information

Shun Lu (Institute of Computing Technology, Chinese Academy of Sciences)
Jixiang Li (Inc. Kuaishou)
Jianchao Tan (Kuaishou Technology)

Research Scientist in Kwai Inc.

Sen Yang (Kuaishou Technology)
Ji Liu (Kwai Inc.)

More from the Same Authors