Timezone: »

[Re] Understanding Self-Supervised Learning Dynamics without Contrastive Pairs
Tobias Höppe · Agnieszka Miszkurka · Dennis Bogatov Wilkman

Tue Dec 06 09:00 AM -- 11:00 AM (PST) @

Self-Supervised learning without contrastive pairs has shown huge success in the recent year. However, understanding why these networks do not collapse despite not using contrastive pairs was not fully understood until very recently. In this work we re-implemented the architectures and pre-training schemes of SimSiam, BYOL, DirectPred and DirectCopy. We investigated the eigenspace alignment hypothesis in DirectPred, by plotting the eigenvalues and eigenspace alignments for both SimSiam and BYOL with and without Symmetric regularization. We also combine the framework of DirectPred with SimCLRv2 in order to explore if any further improvements could be made. We managed to achieve comparable results to the paper of DirectPred in regards to accuracy and the behaviour of symmetry and eigenspace alignment.

Author Information

Tobias Höppe (KTH Stockholm)
Agnieszka Miszkurka (KTH Royal Institute of Science)
Agnieszka Miszkurka

A recent graduate from the master's program in Machine Learning at KTH Royal Institute of Technology. Soon starting as a Software Engineer at Google Zurich.

Dennis Bogatov Wilkman

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors