Timezone: »

 
Momentum Extragradient is Optimal for Games with Cross-Shaped Spectrum
Junhyung Lyle Kim · Gauthier Gidel · Anastasios Kyrillidis · Fabian Pedregosa

@
in OPT 2022: Optimization for Machine Learning »
Event URL: https://openreview.net/forum?id=MbZ3_HSk9-K »
The extragradient method has recently gained a lot of attention, due to its convergence behavior on smooth games. In games, the eigenvalues of the Jacobian of the vector field are distributed on the complex plane, exhibiting more convoluted dynamics compared to minimization. In this work, we take a polynomial-based analysis of the extragradient with momentum for optimizing games with \emph{cross-shaped} spectrum on the complex plane. We show two results: first, the extragradient with momentum exhibits three different modes of convergence based on the hyperparameter setup: when the eigenvalues are distributed $(i)$ on the real line, $(ii)$ both on the real line along with complex conjugates, and $(iii)$ only as complex conjugates. Then, we focus on the case $(ii)$, i.e., when the spectrum of the Jacobian has \emph{cross-shaped} structure, as observed in training generative adversarial networks. For this problem class, we derive the optimal parameters and show that the extragradient with momentum achieves accelerated convergence rate.

Author Information

Junhyung Lyle Kim (Rice University)
Gauthier Gidel (Mila)
Anastasios Kyrillidis (Rice University)
Fabian Pedregosa (Google AI)

More from the Same Authors