Timezone: »

 
Poster
On the distance between two neural networks and the stability of learning
Jeremy Bernstein · Arash Vahdat · Yisong Yue · Ming-Yu Liu

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #310
in Poster Session 2 »

This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is here: https://github.com/jxbz/fromage.

Author Information

Jeremy Bernstein (Caltech)
Arash Vahdat (NVIDIA Research)
Yisong Yue (Caltech)
Ming-Yu Liu (Nvidia Research)

More from the Same Authors