Timezone: »

 
Spotlight
The Statistical Complexity of Early-Stopped Mirror Descent
Tomas Vaskevicius · Varun Kanade · Patrick Rebeschini

Thu Dec 10 07:50 AM -- 08:00 AM (PST) @ Orals & Spotlights: Optimization/Theory
in Orals & Spotlights Track 30: Optimization/Theory »

Recently there has been a surge of interest in understanding implicit regularization properties of iterative gradient-based optimization algorithms. In this paper, we study the statistical guarantees on the excess risk achieved by early-stopped unconstrained mirror descent algorithms applied to the unregularized empirical risk with the squared loss for linear models and kernel methods. By completing an inequality that characterizes convexity for the squared loss, we identify an intrinsic link between offset Rademacher complexities and potential-based convergence analysis of mirror descent methods. Our observation immediately yields excess risk guarantees for the path traced by the iterates of mirror descent in terms of offset complexities of certain function classes depending only on the choice of the mirror map, initialization point, step-size, and the number of iterations. We apply our theory to recover, in a rather clean and elegant manner via rather short proofs, some of the recent results in the implicit regularization literature, while also showing how to improve upon them in some settings.

Keywords: Optimization · Algorithms; Algorithms -> Regression; Algorithms -> Similarity and Distance Learning; Optimization · Combinatorial Optimizatio

Author Information

Tomas Vaskevicius (University of Oxford)
Varun Kanade (University of Oxford)
Patrick Rebeschini (University of Oxford)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors