Timezone: »

 
Poster
Models Out of Line: A Fourier Lens on Distribution Shift Robustness
Sara Fridovich-Keil · Brian Bartoldson · James Diffenderfer · Bhavya Kailkhura · Timo Bremer

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #243
in Poster Session 2 »
Improving the accuracy of deep neural networks on out-of-distribution (OOD) data is critical to an acceptance of deep learning in real world applications. It has been observed that accuracies on in-distribution (ID) versus OOD data follow a linear trend and models that outperform this baseline are exceptionally rare (and referred to as ``effectively robust”). Recently, some promising approaches have been developed to improve OOD robustness: model pruning, data augmentation, and ensembling or zero-shot evaluating large pretrained models. However, there still is no clear understanding of the conditions on OOD data and model properties that are required to observe effective robustness. We approach this issue by conducting a comprehensive empirical study of diverse approaches that are known to impact OOD robustness on a broad range of natural and synthetic distribution shifts of CIFAR-10 and ImageNet. In particular, we view the "effective robustness puzzle" through a Fourier lens and ask how spectral properties of both models and OOD data correlate with OOD robustness. We find this Fourier lens offers some insight into why certain robust models, particularly those from the CLIP family, achieve OOD robustness. However, our analysis also makes clear that no known metric is consistently the best explanation of OOD robustness. Thus, to aid future research into the OOD puzzle, we address the gap in publicly-available models with effective robustness by introducing a set of pretrained CIFAR-10 models---$RobustNets$---with varying levels of OOD robustness.
Keywords: Deep Neural Networks · spectral analysis · CLIP models · OOD robustness · effective robustness
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Sara Fridovich-Keil (UC Berkeley)
Brian Bartoldson (Lawrence Livermore National Laboratory)
James Diffenderfer (Lawrence Livermore National Laboratory)
Bhavya Kailkhura (Lawrence Livermore National Laboratory)
Timo Bremer (Lawrence Livermore National Laboratory)

More from the Same Authors