Poster
in
Workshop: Symmetry and Geometry in Neural Representations (NeurReps)
Practical Structured Riemannian Optimization with Momentum by using Generalized Normal Coordinates
Wu Lin · Valentin Duruisseaux · Melvin Leok · Frank Nielsen · Mohammad Emtiyaz Khan · Mark Schmidt
Keywords: [ Numerical Optimization ] [ Matrix Lie Groups ] [ Riemannian Manifolds ]
Adding momentum into Riemannian optimization is computationally challenging due to the intractable ODEs needed to define the exponential and parallel transport maps. We address these issues for Gaussian Fisher-Rao manifolds by proposing new local coordinates to exploit sparse structures and efficiently approximate the ODEs, which results in a numerically stable update scheme. Our approach extends the structured natural-gradient descent method of Lin et al. (2021a) by incorporating momentum into it and scaling the method for large-scale applications arising in numerical optimization and deep learning