Timezone: »
Recognizing out-of-distribution (OOD) samples is critical for machine learning systems deployed in the open world. The vast majority of OOD detection methods are driven by a single modality (e.g., either vision or language), leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of OOD detection from a single-modal to a multi-modal regime. Particularly, we propose Maximum Concept Matching (MCM), a simple yet effective zero-shot OOD detection method based on aligning visual features with textual concepts. We contribute in-depth analysis and theoretical insights to understand the effectiveness of MCM. Extensive experiments demonstrate that MCM achieves superior performance on a wide variety of real-world tasks. MCM with vision-language features outperforms a common baseline with pure visual features on a hard OOD task with semantically similar classes by 13.1% (AUROC) Code is available at https://github.com/deeplearning-wisc/MCM.
Author Information
Yifei Ming (University of Wisconsin-Madison)
I'm a Ph.D. student at the University of Wisconsin-Madison. I’m broadly interested in trustworthy machine learning and representation learning. Research topics that I am currently focusing on include: out-of-distribution detection, domain generalization, supervised and self-supervised (multi-modal) representation learning. My prior research involves designing efficient algorithms and promoting fundamental understandings to enable reliable open-world learning. (e.g., impact of spurious correlation, sample efficiency, and multi-modality).
Ziyang Cai (University of Wisconsin - Madison)
Jiuxiang Gu (Adobe Research)
Yiyou Sun (University of Wisconsin, Madison)
Wei Li (GOOGLE INC)
Yixuan Li (University of Wisconsin-Madison)
More from the Same Authors
-
2022 Poster: SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learning »
Haobo Wang · Mingxuan Xia · Yixuan Li · Yuren Mao · Lei Feng · Gang Chen · Junbo Zhao -
2022 : Domain Generalization with Nuclear Norm Regularization »
Zhenmei Shi · Yifei Ming · Ying Fan · Frederic Sala · Yingyu Liang -
2023 Poster: Dream the Impossible: Outlier Imagination with Diffusion Models »
Xuefeng Du · Yiyou Sun · Jerry Zhu · Yixuan Li -
2023 Poster: AIMS: All-Inclusive Multi-Level Segmentation »
Lu Qi · Jason Kuen · Weidong Guo · Jiuxiang Gu · Zhe Lin · Bo Du · Yu Xu · Ming-Hsuan Yang -
2023 Poster: A Graph-Theoretic Framework for Understanding Open-World Representation Learning »
Yiyou Sun · Zhenmei Shi · Yixuan Li -
2023 Poster: Learning to Augment Distributions for Out-of-distribution Detection »
Qizhou Wang · Zhen Fang · Yonggang Zhang · Feng Liu · Yixuan Li · Bo Han -
2022 Workshop: Robustness in Sequence Modeling »
Nathan Ng · Haoran Zhang · Vinith Suriyakumar · Chantal Shaib · Kyunghyun Cho · Yixuan Li · Alice Oh · Marzyeh Ghassemi -
2022 Poster: SIREN: Shaping Representations for Detecting Out-of-Distribution Objects »
Xuefeng Du · Gabriel Gozum · Yifei Ming · Yixuan Li -
2022 Poster: Is Out-of-Distribution Detection Learnable? »
Zhen Fang · Yixuan Li · Jie Lu · Jiahua Dong · Bo Han · Feng Liu -
2022 Poster: OpenOOD: Benchmarking Generalized Out-of-Distribution Detection »
Jingkang Yang · Pengyun Wang · Dejian Zou · Zitang Zhou · Kunyuan Ding · WENXUAN PENG · Haoqi Wang · Guangyao Chen · Bo Li · Yiyou Sun · Xuefeng Du · Kaiyang Zhou · Wayne Zhang · Dan Hendrycks · Yixuan Li · Ziwei Liu -
2021 : Uncovering the Deep Unknowns of ImageNet Model: Challenges and Opportunties »
Yixuan Li -
2021 Poster: On the Importance of Gradients for Detecting Distributional Shifts in the Wild »
Rui Huang · Andrew Geng · Yixuan Li -
2021 Poster: Can multi-label classification networks know what they don’t know? »
Haoran Wang · Weitang Liu · Alex Bocchieri · Yixuan Li -
2021 Poster: UniDoc: Unified Pretraining Framework for Document Understanding »
Jiuxiang Gu · Jason Kuen · Vlad I Morariu · Handong Zhao · Rajiv Jain · Nikolaos Barmpalios · Ani Nenkova · Tong Sun -
2021 Poster: ReAct: Out-of-distribution Detection With Rectified Activations »
Yiyou Sun · Chuan Guo · Yixuan Li -
2020 Poster: Energy-based Out-of-distribution Detection »
Weitang Liu · Xiaoyun Wang · John Owens · Yixuan Li -
2020 Poster: Self-Supervised Relationship Probing »
Jiuxiang Gu · Jason Kuen · Shafiq Joty · Jianfei Cai · Vlad I. Morariu · Handong Zhao · Tong Sun