Timezone: »
In this work we propose a novel interpretation of residual networks showing that they can be seen as a collection of many paths of differing length. Moreover, residual networks seem to enable very deep networks by leveraging only the short paths during training. To support this observation, we rewrite residual networks as an explicit collection of paths. Unlike traditional models, paths through residual networks vary in length. Further, a lesion study reveals that these paths show ensemble-like behavior in the sense that they do not strongly depend on each other. Finally, and most surprising, most paths are shorter than one might expect, and only the short paths are needed during training, as longer paths do not contribute any gradient. For example, most of the gradient in a residual network with 110 layers comes from paths that are only 10-34 layers deep. Our results reveal one of the key characteristics that seem to enable the training of very deep networks: Residual networks avoid the vanishing gradient problem by introducing short paths which can carry gradient throughout the extent of very deep networks.
Author Information
Andreas Veit (Cornell University)
Michael J Wilber (Cornell Tech)
Serge Belongie (Cornell University)
More from the Same Authors
-
2021 : Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge »
Jiyang Qi · Yan Gao · Yao Hu · Xinggang Wang · Xiaoyu Liu · Xiang Bai · Serge Belongie · Alan Yuille · Philip Torr · Song Bai -
2021 : An Empirical Study of Pre-trained Models on Out-of-distribution Generalization »
Yaodong Yu · Heinrich Jiang · Dara Bahri · Hossein Mobahi · Seungyeon Kim · Ankit Rawat · Andreas Veit · Yi Ma -
2022 : Effect of mixup Training on Representation Learning »
Arslan Chaudhry · Aditya Menon · Andreas Veit · Sadeep Jayasumana · Srikumar Ramalingam · Sanjiv Kumar -
2021 Poster: Geometry Processing with Neural Fields »
Guandao Yang · Serge Belongie · Bharath Hariharan · Vladlen Koltun -
2019 Poster: Positional Normalization »
Boyi Li · Felix Wu · Kilian Weinberger · Serge Belongie -
2019 Spotlight: Positional Normalization »
Boyi Li · Felix Wu · Kilian Weinberger · Serge Belongie -
2017 : Poster Session 1 and Lunch »
Sumanth Dathathri · Akshay Rangamani · Prakhar Sharma · Aruni RoyChowdhury · Madhu Advani · William Guss · Chulhee Yun · Corentin Hardy · Michele Alberti · Devendra Sachan · Andreas Veit · Takashi Shinozaki · Peter Chin -
2012 Workshop: Human Computation for Science and Computational Sustainability »
Theodoros Damoulas · Thomas Dietterich · Edith Law · Serge Belongie -
2012 Poster: LUCID: Locally Uniform Comparison Image Descriptor »
Andrew M Ziegler · Eric Christiansen · David Kriegman · Serge Belongie -
2010 Oral: The Multidimensional Wisdom of Crowds »
Peter Welinder · Steve Branson · Serge Belongie · Pietro Perona -
2010 Poster: The Multidimensional Wisdom of Crowds »
Peter Welinder · Steve Branson · Serge Belongie · Pietro Perona -
2006 Poster: Learning to Traverse Image Manifolds »
Piotr Dollar · Vincent Rabaud · Serge Belongie