Timezone: »

 
Poster
On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
Chen Liu · Mathieu Salzmann · Tao Lin · Ryota Tomioka · Sabine Süsstrunk

Wed Dec 09 09:00 AM -- 11:00 AM (PST) @ Poster Session 3 #913
in Poster Session 4 »

We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients. Our conclusions are validated by numerical analyses, which show that training under large adversarial budgets impede the escape from suboptimal random initialization, cause non-vanishing gradients and make the models' minima found sharper. Based on these observations, we show that a periodic adversarial scheduling (PAS) strategy can effectively overcome these challenges, yielding better results than vanilla adversarial training while being much less sensitive to the choice of learning rate.

Keywords: Representation Learning · Dialog- or Communication-Based Learning · Applications · Algorithms

Author Information

Chen Liu (EPFL)
Mathieu Salzmann (EPFL)
Tao Lin (EPFL)
Ryota Tomioka (Microsoft Research Cambridge)
Sabine Süsstrunk (EPFL)

More from the Same Authors