Timezone: »

 
Poster
Adaptive Methods for Nonconvex Optimization
Manzil Zaheer · Sashank Reddi · Devendra S Sachan · Satyen Kale · Sanjiv Kumar

Wed Dec 05 02:00 PM -- 04:00 PM (PST) @ Room 210 #55
in Wed Poster Session B »

Adaptive gradient methods that rely on scaling gradients down by the square root of exponential moving averages of past squared gradients, such RMSProp, Adam, Adadelta have found wide application in optimizing the nonconvex problems that arise in deep learning. However, it has been recently demonstrated that such methods can fail to converge even in simple convex optimization settings. In this work, we provide a new analysis of such methods applied to nonconvex stochastic optimization problems, characterizing the effect of increasing minibatch size. Our analysis shows that under this scenario such methods do converge to stationarity up to the statistical limit of variance in the stochastic gradients (scaled by a constant factor). In particular, our result implies that increasing minibatch sizes enables convergence, thus providing a way to circumvent the non-convergence issues. Furthermore, we provide a new adaptive optimization algorithm, Yogi, which controls the increase in effective learning rate, leading to even better performance with similar theoretical guarantees on convergence. Extensive experiments show that Yogi with very little hyperparameter tuning outperforms methods such as Adam in several challenging machine learning tasks.

Keywords: Convex Optimization · Non-Convex Optimization
[ Paper] 

Author Information

Manzil Zaheer (Google)
Sashank Reddi (Google)
Devendra S Sachan (Carnegie Mellon University)
Satyen Kale (Google)
Sanjiv Kumar (Google Research)

More from the Same Authors