Timezone: »
This paper introduces Non-Autonomous Input-Output Stable Network (NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and allows stability to be enforced so that blocks can be unrolled adaptively to a pattern-dependent processing depth. NAIS-Net induces non-trivial, Lipschitz input-output maps, even for an infinite unroll length. We prove that the network is globally asymptotically stable so that for every initial condition there is exactly one input-dependent equilibrium assuming tanh units, and multiple stable equilibria for ReL units. An efficient implementation that enforces the stability under derived conditions for both fully-connected and convolutional layers is also presented. Experimental results show how NAIS-Net exhibits stability in practice, yielding a significant reduction in generalization gap compared to ResNets.
Author Information
Marco Ciccone (Politecnico di Milano)

Marco Ciccone is an ELLIS Postdoctoral Researcher in the VANDAL group at Politecnico di Torino and UCL. His current research interests are in the intersection of meta, continual, and federated learning with a particular focus on modularity and models re-use to scale the training of agents with heterogeneous data and mitigate the effect of catastrophic forgetting and interference across tasks, domains, and devices. He has been NeurIPS Competiton Track co-chair in 2021, 2022 and 2023.
Marco Gallieri (NNAISENSE)
Marco Gallieri is a Research Scientist at NNAISENSE, in Lugano. He received a PhD in Engineering from Sidney Sussex College, the University of Cambridge, in 2014. His PhD Thesis was on LASSO-MPC and is published by Springer. In 2009 he received an MSc in automation engineering from the Universita’ Politecnica delle Marche, in Italy. He wrote his MSc thesis during a visiting term at the National University of Ireland, Maynooth. In 2010 he was a Marie Curie early stage researcher at the Instituto Superior Tecnico in Lisbon working on non-linear control of autonomous underwater vehicles. Before joining NNAISENSE, he spent three years with the McLaren group, where he developed a model based Li-Ion battery management system for the F1 power unit and a prototype for next generation F1 driver-in-the-loop simulator. He then worked as a data scientist in the R&D branch of the group. He’s currently leading the control theory R&D efforts of NNAISENSE. His research interests are at the intersection between control and machine learning and include the study of stability of deep and recurrent neural networks as well as their use in control systems for safety-critical applications.
Jonathan Masci (NNAISENSE)
Christian Osendorfer (NNAISENSE)
Faustino Gomez (NNAISENSE)
More from the Same Authors
-
2021 : Public Information Representation for Adversarial Team Games »
Luca Carminati · Federico Cacciamani · Marco Ciccone · Nicola Gatti -
2022 Competition: NeurIPS 2022 Competition Track: Overview & Results »
Marco Ciccone · Gustavo Stolovitzky · Jake Albrecht -
2021 : Spotlight Talk: Public Information Representation for Adversarial Team Games »
Luca Carminati · Federico Cacciamani · Marco Ciccone · Nicola Gatti -
2021 Demonstration: Demonstrations 4 »
Douwe Kiela · Barbara Caputo · Marco Ciccone -
2021 : Intro »
Marco Ciccone -
2021 : Introduction to Competition Day 4 »
Marco Ciccone -
2021 Competition: Competition Track Day 4: Overviews + Breakout Sessions »
Douwe Kiela · Marco Ciccone · Barbara Caputo -
2021 : Introduction to Competition Day 3 »
Marco Ciccone -
2021 Competition: Competition Track Day 3: Overviews + Breakout Sessions »
Douwe Kiela · Marco Ciccone · Barbara Caputo -
2021 Demonstration: Demonstrations 3 »
Douwe Kiela · Barbara Caputo · Marco Ciccone -
2021 : Intro »
Marco Ciccone -
2021 Demonstration: Demonstrations 2 »
Douwe Kiela · Barbara Caputo · Marco Ciccone -
2021 Competition: Competition Track Day 2: Overviews + Breakout Sessions »
Douwe Kiela · Marco Ciccone · Barbara Caputo -
2021 Competition: Competition Track Day 1: Overviews + Breakout Sessions »
Douwe Kiela · Marco Ciccone · Barbara Caputo -
2021 Demonstration: Demonstrations 1 »
Douwe Kiela · Barbara Caputo · Marco Ciccone -
2020 Poster: Learning to Detect Objects with a 1 Megapixel Event Camera »
Etienne Perot · Pierre de Tournemire · Davide Nitti · Jonathan Masci · Amos Sironi -
2020 Spotlight: Learning to Detect Objects with a 1 Megapixel Event Camera »
Etienne Perot · Pierre de Tournemire · Davide Nitti · Jonathan Masci · Amos Sironi -
2019 : Poster Session »
Ahana Ghosh · Javad Shafiee · Akhilan Boopathy · Alex Tamkin · Theodoros Vasiloudis · Vedant Nanda · Ali Baheri · Paul Fieguth · Andrew Bennett · Guanya Shi · Hao Liu · Arushi Jain · Jacob Tyo · Benjie Wang · Boxiao Chen · Carroll Wainwright · Chandramouli Shama Sastry · Chao Tang · Daniel S. Brown · David Inouye · David Venuto · Dhruv Ramani · Dimitrios Diochnos · Divyam Madaan · Dmitrii Krashenikov · Joel Oren · Doyup Lee · Eleanor Quint · elmira amirloo · Matteo Pirotta · Gavin Hartnett · Geoffroy Dubourg-Felonneau · Gokul Swamy · Pin-Yu Chen · Ilija Bogunovic · Jason Carter · Javier Garcia-Barcos · Jeet Mohapatra · Jesse Zhang · Jian Qian · John Martin · Oliver Richter · Federico Zaiter · Tsui-Wei Weng · Karthik Abinav Sankararaman · Kyriakos Polymenakos · Lan Hoang · mahdieh abbasi · Marco Gallieri · Mathieu Seurin · Matteo Papini · Matteo Turchetta · Matthew Sotoudeh · Mehrdad Hosseinzadeh · Nathan Fulton · Masatoshi Uehara · Niranjani Prasad · Oana-Maria Camburu · Patrik Kolaric · Philipp Renz · Prateek Jaiswal · Reazul Hasan Russel · Riashat Islam · Rishabh Agarwal · Alexander Aldrick · Sachin Vernekar · Sahin Lale · Sai Kiran Narayanaswami · Samuel Daulton · Sanjam Garg · Sebastian East · Shun Zhang · Soheil Dsidbari · Justin Goodwin · Victoria Krakovna · Wenhao Luo · Wesley Chung · Yuanyuan Shi · Yuh-Shyang Wang · Hongwei Jin · Ziping Xu -
2016 Poster: Learning shape correspondence with anisotropic convolutional neural networks »
Davide Boscaini · Jonathan Masci · Emanuele Rodolà · Michael Bronstein -
2014 Poster: Deep Networks with Internal Selective Attention through Feedback Connections »
Marijn F Stollenga · Jonathan Masci · Faustino Gomez · Jürgen Schmidhuber -
2013 Poster: Compete to Compute »
Rupesh K Srivastava · Jonathan Masci · Sohrob Kazerounian · Faustino Gomez · Jürgen Schmidhuber