Timezone: »

 
Poster
Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction
Konstantin Schürholt · Dimche Kostadinov · Damian Borth

Wed Dec 08 12:30 AM -- 02:00 AM (PST) @ Virtual
in Poster Session 3 »

Self-Supervised Learning (SSL) has been shown to learn useful and information-preserving representations. Neural Networks (NNs) are widely applied, yet their weight space is still not fully understood. Therefore, we propose to use SSL to learn hyper-representations of the weights of populations of NNs. To that end, we introduce domain specific data augmentations and an adapted attention architecture. Our empirical evaluation demonstrates that self-supervised representation learning in this domain is able to recover diverse NN model characteristics. Further, we show that the proposed learned representations outperform prior work for predicting hyper-parameters, test accuracy, and generalization gap as well as transfer to out-of-distribution settings.

Keywords: Deep Learning · Self-Supervised Learning · Representation Learning

Author Information

Konstantin Schürholt (University of St. Gallen)
Dimche Kostadinov (University of Geneva)
Damian Borth (University of St.Gallen (HSG))

More from the Same Authors