Timezone: »

 
Poster
Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis
Wuyang Chen · Wei Huang · Xinyu Gong · Boris Hanin · Zhangyang Wang

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #121
in Poster Session 4 »

Advanced deep neural networks (DNNs), designed by either human or AutoML algorithms, are growing increasingly complex. Diverse operations are connected by complicated connectivity patterns, e.g., various types of skip connections. Those topological compositions are empirically effective and observed to smooth the loss landscape and facilitate the gradient flow in general. However, it remains elusive to derive any principled understanding of their effects on the DNN capacity or trainability, and to understand why or in which aspect one specific connectivity pattern is better than another. In this work, we theoretically characterize the impact of connectivity patterns on the convergence of DNNs under gradient descent training in fine granularity. By analyzing a wide network's Neural Network Gaussian Process (NNGP), we are able to depict how the spectrum of an NNGP kernel propagates through a particular connectivity pattern, and how that affects the bound of convergence rates. As one practical implication of our results, we show that by a simple filtration of "unpromising" connectivity patterns, we can trim down the number of models to evaluate, and significantly accelerate the large-scale neural architecture search without any overhead.

Keywords: convergence · Neural Network Gaussian Process · Neural Network Architecture
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Wuyang Chen (University of Texas, Austin)
Wei Huang (RIKEN AIP)

32/255-271 Anzac Parade Kingsford NSW 2032

Xinyu Gong (University of Texas, Austin)
Boris Hanin (Princeton)
Zhangyang Wang (University of Texas at Austin)

More from the Same Authors