Timezone: »

 
Poster
When Do Flat Minima Optimizers Work?
Jean Kaddour · Linqing Liu · Ricardo Silva · Matt Kusner

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #124
in Poster Session 1 »

Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.

Keywords: Deep Learning · Optimization · flatness
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Jean Kaddour (University College London)
Linqing Liu (University College London)
Ricardo Silva (University College London)
Matt Kusner (University College London)

More from the Same Authors