Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients

It has been widely observed that Adam can remain stable even when the objective function deviates significantly from global smoothness. However, under the generalized smoothness framework, existing theoretical analyses typically rely on strong tail assumptions on stochastic gradients, such as almost-sure boundedness or sub-Gaussianity. Whether one can establish the convergence of Adam on generalized smooth objectives under only second moment information on the stochastic gradients, without imposing such strong concentration assumptions, was explicitly identified as an important open direction by \citet{li2023convex}. This paper gives an affirmative answer to this question under fairly general conditions. Specifically, we prove that such strong tail assumptions are not necessary. We show that the key mechanism by which Adam remains stable and achieves convergence under the (L0)--(Lp) generalized smoothness condition is the self-normalization effect induced by its adaptive coordinate-wise scaling. Based on this mechanism, we prove that even under a very general stochastic-gradient condition, namely a generalized second moment ABC condition that provides only second moment information, the stochastic trajectory of Adam remains in a locally well-behaved smoothness region with stretched-exponential tail decay. As a consequence, we establish high-probability convergence rate guarantees over the full range (p<2), with a confidence dependence of order (\delta^{-1/2}), while the required stepsize calibration depends on (\delta) only through polylogarithmic factors in (\polylog(1/\delta)). Furthermore, we construct a hard instance proving that, under only second-moment information on the stochastic gradients, this (\delta^{-1/2})-type confidence dependence is sharp. Finally, in the more favorable regime (p<1), we combine the above trajectory control with polynomial-growth estimates on rare events to further obtain convergence rate guarantees in expectation.