Timezone: »

 
Poster
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
Manli Shu · Weili Nie · De-An Huang · Zhiding Yu · Tom Goldstein · Anima Anandkumar · Chaowei Xiao

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #633
in Poster Session 5 »

Pre-trained vision-language models (e.g., CLIP) have shown promising zero-shot generalization in many downstream tasks with properly designed text prompts. Instead of relying on hand-engineered prompts, recent works learn prompts using the training data from downstream tasks. While effective, training on domain-specific data reduces a model's generalization capability to unseen new domains. In this work, we propose test-time prompt tuning (TPT), a method that can learn adaptive prompts on the fly with a single test sample. TPT optimizes the prompt by minimizing the entropy with confidence selection so that the model has consistent predictions across different augmented views of each test sample. In evaluating generalization to natural distribution shifts, TPT improves the zero-shot top-1 accuracy of CLIP by 3.6\% on average, surpassing previous prompt tuning approaches that require additional task-specific training data. In evaluating cross-dataset generalization with unseen categories, TPTperforms on par with the state-of-the-art approaches that use additional training data.

Keywords: generalization · Out-of-Distribution Robustness · Prompt Tuning · Foundation Model
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Manli Shu (University of Maryland, College Park)
Weili Nie (NVIDIA)
De-An Huang (NVIDIA)
Zhiding Yu (NVIDIA)
Tom Goldstein (University of Maryland)
Anima Anandkumar (NVIDIA / Caltech)
Chaowei Xiao (ASU/NVIDIA)

I am Chaowei Xiao, a third year PhD student in CSE Department, University of Michigan, Ann Arbor. My advisor is Professor Mingyan Liu . I obtained my bachelor's degree in School of Software from Tsinghua University in 2015, advised by Professor Yunhao Liu, Professor Zheng Yang and Dr. Lei Yang. I was also a visiting student at UC Berkeley in 2018, advised by Professor Dawn Song and Professor Bo Li. My research interest includes adversarial machine learning.

More from the Same Authors