Poster
Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training
Atli Kosson · Bettina Messmer · Martin Jaggi
East Exhibit Hall A-C #1810
Abstract:
Learning Rate Warmup is a popular heuristic for training neural networks, especially at larger batch sizes, despite limited understanding of its benefits. Warmup decreases the update size early in training by using lower values for the learning rate . In this work we argue that warmup benefits training by keeping the overall size of limited, counteracting large initial values of . Focusing on small-scale GPT training with AdamW/Lion, we explore the following question: *Why and by which criteria are early updates too large?* We analyze different metrics for the update size including the -norm, resulting directional change, and impact on the representations of the network, providing a new perspective on warmup. In particular, we find that warmup helps counteract large angular updates as well as a limited critical batch size early in training. Finally, we show that the need for warmup can be significantly reduced or eliminated by modifying the optimizer to explicitly normalize based on the aforementioned metrics.
Chat is not available.