Timezone: »

 
Poster
NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
Marco Ciccone · Marco Gallieri · Jonathan Masci · Christian Osendorfer · Faustino Gomez

Tue Dec 04 07:45 AM -- 09:45 AM (PST) @ Room 517 AB #132
in Tue Poster Session A »

This paper introduces Non-Autonomous Input-Output Stable Network (NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and allows stability to be enforced so that blocks can be unrolled adaptively to a pattern-dependent processing depth. NAIS-Net induces non-trivial, Lipschitz input-output maps, even for an infinite unroll length. We prove that the network is globally asymptotically stable so that for every initial condition there is exactly one input-dependent equilibrium assuming tanh units, and multiple stable equilibria for ReL units. An efficient implementation that enforces the stability under derived conditions for both fully-connected and convolutional layers is also presented. Experimental results show how NAIS-Net exhibits stability in practice, yielding a significant reduction in generalization gap compared to ResNets.

Keywords: Deep Learning · Representation Learning · Dynamical Systems · Control Theory
[ Paper] 

Author Information

Marco Ciccone (Politecnico di Milano)
Marco Ciccone

Marco Ciccone is an ELLIS Postdoctoral Researcher in the VANDAL group at Politecnico di Torino and UCL. His current research interests are in the intersection of meta, continual, and federated learning with a particular focus on modularity and models re-use to scale the training of agents with heterogeneous data and mitigate the effect of catastrophic forgetting and interference across tasks, domains, and devices. He has been NeurIPS Competiton Track co-chair in 2021, 2022 and 2023.

Marco Gallieri (NNAISENSE)

Marco Gallieri is a Research Scientist at NNAISENSE, in Lugano. He received a PhD in Engineering from Sidney Sussex College, the University of Cambridge, in 2014. His PhD Thesis was on LASSO-MPC and is published by Springer.  In 2009 he received an MSc in automation engineering from the Universita’ Politecnica delle Marche, in Italy. He wrote his MSc thesis during a visiting term at the National University of Ireland, Maynooth.  In 2010 he was a Marie Curie early stage researcher at the Instituto Superior Tecnico in Lisbon working on non-linear control of autonomous underwater vehicles. Before joining NNAISENSE, he spent three years with the McLaren group, where he developed a model based Li-Ion battery management system for the F1 power unit and a prototype for next generation F1 driver-in-the-loop simulator. He then worked as a data scientist in the R&D branch of the group. He’s currently leading the control theory R&D efforts of NNAISENSE. His research interests are at the intersection between control and machine learning and include the study of stability of deep and recurrent neural networks as well as their use in control systems for safety-critical applications.

Jonathan Masci (NNAISENSE)
Christian Osendorfer (NNAISENSE)
Faustino Gomez (NNAISENSE)

More from the Same Authors