Timezone: »

 
Poster
Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach
Peng Mi · Li Shen · Tianhe Ren · Yiyi Zhou · Xiaoshuai Sun · Rongrong Ji · Dacheng Tao

@
Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation,~\emph{i.e.}, double the overhead of common optimizers like Stochastic Gradient Descent~(SGD). In this paper, we propose an efficient and effective training scheme coined as Sparse SAM (SSAM), which achieves sparse perturbation by a binary mask. To obtain the sparse mask, we provide two solutions which are based on Fisher information and dynamic sparse training, respectively. In addition, we theoretically prove that SSAM can converge at the same rate as SAM,~\emph{i.e.}, $O(\log T/\sqrt{T})$. Sparse SAM not only has the potential for training acceleration but also smooths the loss landscape effectively. Extensive experimental results on CIFAR10, CIFAR100, and ImageNet-1K confirm the superior efficiency of our method to SAM, and the performance is preserved or even better with a perturbation of merely 50\% sparsity. Code is available at \url{https://github.com/Mi-Peng/Sparse-Sharpness-Aware-Minimization}.
Keywords: dynamic sparse training · Fisher information · Sharpness-Aware Minimization
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Peng Mi (Xiamen University)
Li Shen (Tencent AI Lab)
Tianhe Ren (Xiamen University)
Yiyi Zhou (Xiamen University)
Xiaoshuai Sun (Xiamen University)
Rongrong Ji (Xiamen University, China)
Dacheng Tao (University of Technology, Sydney)

More from the Same Authors