Skip to yearly menu bar Skip to main content

Workshop: OPT 2023: Optimization for Machine Learning

How to Guess a Gradient

Utkarsh Singhal · Brian Cheung · Kartik Chandra · Jonathan Ragan-Kelley · Josh Tenenbaum · Tomaso Poggio · Stella X. Yu


What can you say about the gradient of a neural network without \emph{computing a loss} or \emph{knowing the label?} This may sound like a strange question: surely the answer is ``very little.'' However, in this paper, we show that gradients are more structured than previously thought. They lie in a predictable low-dimensional subspace which depends on the network architecture and incoming features.Exploiting this structure can significantly improve gradient-free optimization schemes based on directional derivatives, which until now have struggled to scale beyond small networks trained on MNIST. We study how to narrow the gap in optimization performance between methods that calculate exact gradients and those that use directional derivatives, demonstrate new phenomena that occur when using these methods, and highlight new challenges in scaling these methods.

Chat is not available.