Timezone: »

 
Poster
Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning
Timo Milbich · Karsten Roth · Samarth Sinha · Ludwig Schmidt · Marzyeh Ghassemi · Bjorn Ommer

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @
in Poster Session 6 »

Deep Metric Learning (DML) aims to find representations suitable for zero-shot transfer to a priori unknown test distributions. However, common evaluation protocols only test a single, fixed data split in which train and test classes are assigned randomly. More realistic evaluations should consider a broad spectrum of distribution shifts with potentially varying degree and difficulty.In this work, we systematically construct train-test splits of increasing difficulty and present the ooDML benchmark to characterize generalization under out-of-distribution shifts in DML. ooDML is designed to probe the generalization performance on much more challenging, diverse train-to-test distribution shifts. Based on our new benchmark, we conduct a thorough empirical analysis of state-of-the-art DML methods. We find that while generalization tends to consistently degrade with difficulty, some methods are better at retaining performance as the distribution shift increases. Finally, we propose few-shot DML as an efficient way to consistently improve generalization in response to unknown test shifts presented in ooDML.

Keywords: Representation Learning · Transfer Learning

Author Information

Timo Milbich (LMU Munich & Heidelberg University)
Karsten Roth (University of Tuebingen)
Samarth Sinha (University of Toronto, Vector Institute)
Ludwig Schmidt (University of Washington)
Marzyeh Ghassemi (University of Toronto, Vector Institute)
Bjorn Ommer (Heidelberg University)
Bjorn Ommer

Björn Ommer is a full professor at University of Munich where he is heading the Computer Vision & Learning Group. Before he was a full professor in the department of mathematics and computer science at Heidelberg University and a co-director of its Interdisciplinary Center for Scientific Computing. He received his diploma in computer science from University of Bonn, his PhD from ETH Zurich, and he was a postdoc at UC Berkeley. Björn serves as an associate editor for IEEE T-PAMI. His research interests include semantic scene understanding and retrieval, generative AI and visual synthesis, self-supervised metric and representation learning, and explainable AI. Moreover, he is applying this basic research in interdisciplinary projects within neuroscience and the digital humanities. His group has published a series of generative approaches, including "VQGAN" and "Stable Diffusion", which are now democratizing the creation of visual content and have already opened up an abundance of new directions in research, industry, the media, and beyond.

More from the Same Authors