Timezone: »
Differentiable architecture search (DARTS) has been a popular one-shot paradigm for NAS due to its high efficiency. It introduces trainable architecture parameters to represent the importance of candidate operations and proposes first/second-order approximation to estimate their gradients, making it possible to solve NAS by gradient descent algorithm. However, our in-depth empirical results show that the approximation often distorts the loss landscape, leading to the biased objective to optimize and, in turn, inaccurate gradient estimation for architecture parameters. This work turns to zero-order optimization and proposes a novel NAS scheme, called ZARTS, to search without enforcing the above approximation. Specifically, three representative zero-order optimization methods are introduced: RS, MGS, and GLD, among which MGS performs best by balancing the accuracy and speed. Moreover, we explore the connections between RS/MGS and gradient descent algorithm and show that our ZARTS can be seen as a robust gradient-free counterpart to DARTS. Extensive experiments on multiple datasets and search spaces show the remarkable performance of our method. In particular, results on 12 benchmarks verify the outstanding robustness of ZARTS, where the performance of DARTS collapses due to its known instability issue. Also, we search on the search space of DARTS to compare with peer methods, and our discovered architecture achieves 97.54\% accuracy on CIFAR-10 and 75.7\% top-1 accuracy on ImageNet. Finally, we combine our ZARTS with three orthogonal variants of DARTS for faster search speed and better performance. Source code will be made publicly available at: \url{https://github.com/vicFigure/ZARTS}.
Author Information
Xiaoxing Wang (Shanghai Jiao Tong University)
Wenxuan Guo (Shanghai Jiao Tong University)
Jianlin Su (Shenzhen Zhuiyi Technology Co., Ltd.)
Xiaokang Yang (Shanghai Jiao Tong University)
Junchi Yan (Shanghai Jiao Tong University)
More from the Same Authors
-
2022 Poster: Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models »
Minting Pan · Xiangming Zhu · Yunbo Wang · Xiaokang Yang -
2022 Poster: Improving Generative Adversarial Networks via Adversarial Learning in Latent Space »
Yang Li · Yichuan Mo · Liangliang Shi · Junchi Yan -
2022 Poster: Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop »
Weixia Zhang · Dingquan Li · Xiongkuo Min · Guangtao Zhai · Guodong Guo · Xiaokang Yang · Kede Ma -
2022 Poster: Learning Substructure Invariance for Out-of-Distribution Molecular Representations »
Nianzu Yang · Kaipeng Zeng · Qitian Wu · Xiaosong Jia · Junchi Yan -
2023 Poster: Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity »
Zhanpeng Zhou · Yongyi Yang · Xiaojiang Yang · Junchi Yan · Wei Hu -
2023 Poster: H2RBox-v2: Incorporating Symmetry for Boosting Horizontal Box Supervised Oriented Object Detection »
Yi Yu · Xue Yang · Qingyun Li · Yue Zhou · Feipeng Da · Junchi Yan -
2023 Poster: Relative Entropic Optimal Transport: a (Prior-aware) Matching Perspective to (Unbalanced) Classification »
Liangliang Shi · Haoyu Zhen · Gu Zhang · Junchi Yan -
2023 Poster: HubRouter: Learning Global Routing via Hub Generation and Pin-hub Connection »
Xingbo Du · Chonghua Wang · Ruizhe Zhong · Junchi Yan -
2023 Poster: Simplifying and Empowering Transformers for Large-Graph Representations »
Qitian Wu · Wentao Zhao · Chenxiao Yang · Hengrui Zhang · Fan Nie · Haitian Jiang · Yatao Bian · Junchi Yan -
2023 Poster: From Distribution Learning in Training to Gradient Search in Testing for Combinatorial Optimization »
Yang Li · Jinpei Guo · Runzhong Wang · Junchi Yan -
2023 Poster: NeRF-IBVS: Visual Servo Based on NeRF for Visual Localization and Navigation »
Yuanze Wang · Yichao Yan · Dianxi Shi · Wenhan Zhu · Jianqiang Xia · Tan Jeff · Songchang Jin · KE GAO · XIAOBO LI · Xiaokang Yang -
2023 Poster: OpenLane-V2: A Topology Reasoning Benchmark for Scene Understanding in Autonomous Driving »
Huijie Wang · Tianyu Li · Yang Li · Li Chen · Chonghao Sima · Zhenbo Liu · Bangjun Wang · Peijin Jia · Yuting Wang · Shengyin Jiang · Feng Wen · Hang Xu · Ping Luo · Junchi Yan · Wei Zhang · Hongyang Li -
2022 Spotlight: Lightning Talks 5A-3 »
Minting Pan · Xiang Chen · Wenhan Huang · Can Chang · Zhecheng Yuan · Jianzhun Shao · Yushi Cao · Peihao Chen · Ke Xue · Zhengrong Xue · Zhiqiang Lou · Xiangming Zhu · Lei Li · Zhiming Li · Kai Li · Jiacheng Xu · Dongyu Ji · Ni Mu · Kun Shao · Tianpei Yang · Kunyang Lin · Ningyu Zhang · Yunbo Wang · Lei Yuan · Bo Yuan · Hongchang Zhang · Jiajun Wu · Tianze Zhou · Xueqian Wang · Ling Pan · Yuhang Jiang · Xiaokang Yang · Xiaozhuan Liang · Hao Zhang · Weiwen Hu · Miqing Li · YAN ZHENG · Matthew Taylor · Huazhe Xu · Shumin Deng · Chao Qian · YI WU · Shuncheng He · Wenbing Huang · Chuanqi Tan · Zongzhang Zhang · Yang Gao · Jun Luo · Yi Li · Xiangyang Ji · Thomas Li · Mingkui Tan · Fei Huang · Yang Yu · Huazhe Xu · Dongge Wang · Jianye Hao · Chuang Gan · Yang Liu · Luo Si · Hangyu Mao · Huajun Chen · Jianye Hao · Jun Wang · Xiaotie Deng -
2022 Spotlight: Lightning Talks 5B-3 »
Yanze Wu · Jie Xiao · Nianzu Yang · Jieyi Bi · Jian Yao · Yiting Chen · Qizhou Wang · Yangru Huang · Yongqiang Chen · Peixi Peng · Yuxin Hong · Xintao Wang · Feng Liu · Yining Ma · Qibing Ren · Xueyang Fu · Yonggang Zhang · Kaipeng Zeng · Jiahai Wang · GEN LI · Yonggang Zhang · Qitian Wu · Yifan Zhao · Chiyu Wang · Junchi Yan · Feng Wu · Yatao Bian · Xiaosong Jia · Ying Shan · Zhiguang Cao · Zheng-Jun Zha · Guangyao Chen · Tianjun Xiao · Han Yang · Jing Zhang · Jinbiao Chen · MA Kaili · Yonghong Tian · Junchi Yan · Chen Gong · Tong He · Binghui Xie · Yuan Sun · Francesco Locatello · Tongliang Liu · Yeow Meng Chee · David P Wipf · Tongliang Liu · Bo Han · Bo Han · Yanwei Fu · James Cheng · Zheng Zhang -
2022 Spotlight: Lightning Talks 5A-2 »
Qiang LI · Zhiwei Xu · Jia-Qi Yang · Thai Hung Le · Haoxuan Qu · Yang Li · Artyom Sorokin · Peirong Zhang · Mira Finkelstein · Nitsan levy · Chung-Yiu Yau · dapeng li · Thommen Karimpanal George · De-Chuan Zhan · Nazar Buzun · Jiajia Jiang · Li Xu · Yichuan Mo · Yujun Cai · Yuliang Liu · Leonid Pugachev · Bin Zhang · Lucy Liu · Hoi-To Wai · Liangliang Shi · Majid Abdolshah · Yoav Kolumbus · Lin Geng Foo · Junchi Yan · Mikhail Burtsev · Lianwen Jin · Yuan Zhan · Dung Nguyen · David Parkes · Yunpeng Baiia · Jun Liu · Kien Do · Guoliang Fan · Jeffrey S Rosenschein · Sunil Gupta · Sarah Keren · Svetha Venkatesh -
2022 Spotlight: Improving Generative Adversarial Networks via Adversarial Learning in Latent Space »
Yang Li · Yichuan Mo · Liangliang Shi · Junchi Yan -
2022 Spotlight: Iso-Dream: Isolating Noncontrollable Visual Dynamics in World Models »
Minting Pan · Xiangming Zhu · Yunbo Wang · Xiaokang Yang -
2022 Spotlight: Learning Substructure Invariance for Out-of-Distribution Molecular Representations »
Nianzu Yang · Kaipeng Zeng · Qitian Wu · Xiaosong Jia · Junchi Yan -
2022 Spotlight: Rethinking and Improving Robustness of Convolutional Neural Networks: a Shapley Value-based Approach in Frequency Domain »
Yiting Chen · Qibing Ren · Junchi Yan -
2022 Spotlight: Lightning Talks 4B-2 »
Artem Moskalev · Weixia Zhang · Vudtiwat Ngampruetikorn · Anna Sepliarskaia · Dingquan Li · David Schwab · Ivan Sosnovik · Xiongkuo Min · Arnold Smeulders · Guangtao Zhai · Guodong Guo · Xiaokang Yang · Kede Ma -
2022 Spotlight: Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop »
Weixia Zhang · Dingquan Li · Xiongkuo Min · Guangtao Zhai · Guodong Guo · Xiaokang Yang · Kede Ma -
2022 Spotlight: NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification »
Qitian Wu · Wentao Zhao · Zenan Li · David P Wipf · Junchi Yan -
2022 Spotlight: Lightning Talks 1B-1 »
Qitian Wu · Runlin Lei · Rongqin Chen · Luca Pinchetti · Yangze Zhou · Abhinav Kumar · Hans Hao-Hsun Hsu · Wentao Zhao · Chenhao Tan · Zhen Wang · Shenghui Zhang · Yuesong Shen · Tommaso Salvatori · Gitta Kutyniok · Zenan Li · Amit Sharma · Leong Hou U · Yordan Yordanov · Christian Tomani · Bruno Ribeiro · Yaliang Li · David P Wipf · Daniel Cremers · Bolin Ding · Beren Millidge · Ye Li · Yuhang Song · Junchi Yan · Zhewei Wei · Thomas Lukasiewicz -
2022 Poster: NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification »
Qitian Wu · Wentao Zhao · Zenan Li · David P Wipf · Junchi Yan -
2022 Poster: Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks »
Chenxiao Yang · Qitian Wu · Junchi Yan -
2022 Poster: Adv-Attribute: Inconspicuous and Transferable Adversarial Attack on Face Recognition »
Shuai Jia · Bangjie Yin · Taiping Yao · Shouhong Ding · Chunhua Shen · Xiaokang Yang · Chao Ma -
2022 Poster: Rethinking and Improving Robustness of Convolutional Neural Networks: a Shapley Value-based Approach in Frequency Domain »
Yiting Chen · Qibing Ren · Junchi Yan -
2022 Poster: GraphDE: A Generative Framework for Debiased Learning and Out-of-Distribution Detection on Graphs »
Zenan Li · Qitian Wu · Fan Nie · Junchi Yan -
2022 Poster: The Policy-gradient Placement and Generative Routing Neural Networks for Chip Design »
Ruoyu Cheng · Xianglong Lyu · Yang Li · Junjie Ye · Jianye Hao · Junchi Yan -
2022 Poster: Dual-discriminative Graph Neural Network for Imbalanced Graph-level Anomaly Detection »
GE ZHANG · Zhenyu Yang · Jia Wu · Jian Yang · Shan Xue · Hao Peng · Jianlin Su · Chuan Zhou · Quan Z. Sheng · Leman Akoglu · Charu Aggarwal -
2022 Poster: Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment »
Chenxiao Yang · Qitian Wu · Qingsong Wen · Zhiqiang Zhou · Liang Sun · Junchi Yan -
2022 Poster: Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline »
Penghao Wu · Xiaosong Jia · Li Chen · Junchi Yan · Hongyang Li · Yu Qiao -
2022 Poster: GraphQNTK: Quantum Neural Tangent Kernel for Graph Data »
Yehui Tang · Junchi Yan -
2022 Poster: CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation »
Yicong Peng · Yichao Yan · Shengqi Liu · Yuhao Cheng · Shanyan Guan · Bowen Pan · Guangtao Zhai · Xiaokang Yang -
2021 Poster: A Bi-Level Framework for Learning to Solve Combinatorial Optimization on Graphs »
Runzhong Wang · Zhigang Hua · Gan Liu · Jiayi Zhang · Junchi Yan · Feng Qi · Shuang Yang · Jun Zhou · Xiaokang Yang -
2020 Poster: Graduated Assignment for Joint Multi-Graph Matching and Clustering with Application to Unsupervised Graph Matching Network Learning »
Runzhong Wang · Junchi Yan · Xiaokang Yang -
2020 Poster: The Diversified Ensemble Neural Network »
Shaofeng Zhang · Meng Liu · Junchi Yan -
2020 Poster: Adversarial Learning for Robust Deep Clustering »
Xu Yang · Cheng Deng · Kun Wei · Junchi Yan · Wei Liu -
2019 Poster: Learning Latent Process from High-Dimensional Event Sequences via Efficient Sampling »
Qitian Wu · Zixuan Zhang · Xiaofeng Gao · Junchi Yan · Guihai Chen -
2018 Poster: Generalizing Graph Matching beyond Quadratic Assignment Model »
Tianshu Yu · Junchi Yan · Yilin Wang · Wei Liu · baoxin Li -
2018 Poster: Video Prediction via Selective Sampling »
Jingwei Xu · Bingbing Ni · Xiaokang Yang -
2017 Poster: Wasserstein Learning of Deep Generative Point Process Models »
Shuai Xiao · Mehrdad Farajtabar · Xiaojing Ye · Junchi Yan · Xiaokang Yang · Le Song · Hongyuan Zha