Timezone: »

 
Poster
Towards Understanding the Importance of Shortcut Connections in Residual Networks
Tianyi Liu · Minshuo Chen · Mo Zhou · Simon Du · Enlu Zhou · Tuo Zhao

Thu Dec 12 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #171
in Deep Learning -- Optimization for Deep Networks »

Residual Network (ResNet) is undoubtedly a milestone in deep learning. ResNet is equipped with shortcut connections between layers, and exhibits efficient training using simple first order algorithms. Despite of the great empirical success, the reason behind is far from being well understood. In this paper, we study a two-layer non-overlapping convolutional ResNet. Training such a network requires solving a non-convex optimization problem with a spurious local optimum. We show, however, that gradient descent combined with proper normalization, avoids being trapped by the spurious local optimum, and converges to a global optimum in polynomial time, when the weight of the first layer is initialized at 0, and that of the second layer is initialized arbitrarily in a ball. Numerical experiments are provided to support our theory.

Keywords: Deep Learning · Optimization for Deep Networks · Deep Learning
[ Paper]  [ Poster] 

Author Information

Tianyi Liu (Georgia Institute of Technolodgy)
Minshuo Chen (Georgia Tech)
Mo Zhou (Duke University)
Simon Du (Institute for Advanced Study)
Enlu Zhou (Georgia Institute of Technology)
Tuo Zhao (Gatech)

More from the Same Authors