Timezone: »

 
Adaptive Optimization with Examplewise Gradients
Julius Kunze · James Townsend · David Barber

@
in OPT 2021: Optimization for Machine Learning »

We propose a new, more general approach to the design of stochastic gradient-based optimization methods for machine learning. In this new framework, optimizers assume access to a batch of gradient estimates per iteration, rather than a single estimate. This better reflects the information that is actually available in typical machine learning setups. To demonstrate the usefulness of this generalized approach, we develop Eve, an adaptation of the Adam optimizer which uses examplewise gradients to obtain more accurate second-moment estimates. We provide preliminary experiments, without hyperparameter tuning, which show that the new optimizer slightly outperforms Adam on a small scale benchmark and performs the same or worse on larger scale benchmarks. Further work is needed to refine the algorithm and tune hyperparameters.

Author Information

Julius Kunze (University College London)
James Townsend (University College London)
David Barber (University College London)

More from the Same Authors