Timezone: »

 
Poster
Towards Better Generalization of Adaptive Gradient Methods
Yingxue Zhou · Belhal Karimi · Jinxing Yu · Zhiqiang Xu · Ping Li

Wed Dec 09 09:00 AM -- 11:00 AM (PST) @ Poster Session 3 #810
in Poster Session 4 »

Adaptive gradient methods such as AdaGrad, RMSprop and Adam have been optimizers of choice for deep learning due to their fast training speed. However, it was recently observed that their generalization performance is often worse than that of SGD for over-parameterized neural networks. While new algorithms such as AdaBound, SWAT, and Padam were proposed to improve the situation, the provided analyses are only committed to optimization bounds for the training objective, leaving critical generalization capacity unexplored. To close this gap, we propose \textit{\textbf{S}table \textbf{A}daptive \textbf{G}radient \textbf{D}escent} (\textsc{SAGD}) for nonconvex optimization which leverages differential privacy to boost the generalization performance of adaptive gradient methods. Theoretical analyses show that \textsc{SAGD} has high-probability convergence to a population stationary point. We further conduct experiments on various popular deep learning tasks and models. Experimental results illustrate that \textsc{SAGD} is empirically competitive and often better than baselines.

Author Information

Yingxue Zhou (University of Minnesota)
Belhal Karimi (Baidu Research)
Jinxing Yu (Baidu Research)
Zhiqiang Xu (Baidu Inc.)
Ping Li (Baidu Research USA)

More from the Same Authors