Timezone: »

 
Poster
Patching open-vocabulary models by interpolating weights
Gabriel Ilharco · Mitchell Wortsman · Samir Yitzhak Gadre · Shuran Song · Hannaneh Hajishirzi · Simon Kornblith · Ali Farhadi · Ludwig Schmidt

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #611
in Poster Session 6 »

Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks. However, there are still settings where their zero-shot performance is far from optimal. We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate. Towards this goal, we introduce PAINT, a patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning on a task to be patched. On nine tasks where zero-shot CLIP performs poorly, PAINT increases accuracy by 15 to 60 percentage points while preserving accuracy on ImageNet within one percentage point of the zero-shot model. PAINT also allows a single model to be patched on multiple tasks and improves with model scale. Furthermore, we identify cases of broad transfer, where patching on one task increases accuracy on other tasks even when the tasks have disjoint classes. Finally, we investigate applications beyond common benchmarks such as counting or reducing the impact of typographic attacks on CLIP. Our findings demonstrate that it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy without re-training them from scratch.

Keywords: CLIP · vision-and-language · zero-shot models · image-text models · model patching · model editing · open-vocabulary models
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Gabriel Ilharco (Department of Computer Science, University of Washington)
Mitchell Wortsman (University of Washington, Allen Institute for Artificial Intelligence)
Samir Yitzhak Gadre (Columbia University)
Shuran Song (Columbia University)
Hannaneh Hajishirzi (University of Washington)
Simon Kornblith (Google Brain)
Ali Farhadi (University of Washington, Allen Institute for Artificial Intelligence)
Ludwig Schmidt (University of Washington)

More from the Same Authors