Timezone: »

 
Poster
On Margin Maximization in Linear and ReLU Networks
Gal Vardi · Ohad Shamir · Nati Srebro

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #811
in Poster Session 6 »

The implicit bias of neural networks has been extensively studied in recent years. Lyu and Li (2019) showed that in homogeneous networks trained with the exponential or the logistic loss, gradient flow converges to a KKT point of the max margin problem in parameter space. However, that leaves open the question of whether this point will generally be an actual optimum of the max margin problem. In this paper, we study this question in detail, for several neural network architectures involving linear and ReLU activations. Perhaps surprisingly, we show that in many cases, the KKT point is not even a local optimum of the max margin problem. On the flip side, we identify multiple settings where a local or global optimum can be guaranteed.

Keywords: implicit bias · Homogeneous neural networks · Maximum margin
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Gal Vardi (TTI-Chicago)
Ohad Shamir (Weizmann Institute of Science)
Nati Srebro (TTI-Chicago)

More from the Same Authors