Timezone: »

 
Poster
Path-SGD: Path-Normalized Optimization in Deep Neural Networks
Behnam Neyshabur · Russ Salakhutdinov · Nati Srebro

Tue Dec 08 04:00 PM -- 08:59 PM (PST) @ 210 C #33
in Poster Session »

We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with respect to a path-wise regularizer related to max-norm regularization. Path-SGD is easy and efficient to implement and leads to empirical gains over SGD and AdaGrad.

[ PDF] 

Author Information

Behnam Neyshabur (TTI Chicago)
Russ Salakhutdinov (University of Toronto)
Nati Srebro (Toyota Technological Institute at Chicago)

More from the Same Authors