Zeroth-Order Sharpness-Aware Learning with Exponential Tilting

Classic zeroth-order optimization approaches typically optimize for a smoothed version of the original function, i.e., the expected objective under randomly perturbed model parameters. This can be interpreted as encouraging the averaged loss values in the perturbation set to be small. However, popular sharpness-aware minimization objectives typically focus on the largest loss within the neighborhood to arrive at flat minima more effectively. In this work, we connect zeroth-order optimization (and its corresponding objectives) with SAM approaches explicitly, through an exponential tilting objective that provides a smooth transition between the $\texttt{average}$- and the $\texttt{max}$-loss formulations. We explore new zeroth-order algorithms to solve a $\textit{soft}$ SAM objective parameterized by a tilting parameter $t$. We theoretically analyze the sharpness of the tilted objective under different perturbations. Practically, our approach can be used as a gradient-free and memory-efficient alternative to SAM variants, and it achieves better generalization compared to vanilla zeroth-order baselines on a wide range of models and downstream tasks, including classification, multiple choice QA, and language generation.