Timezone: »

 
Poster
Visual Prompting via Image Inpainting
Amir Bar · Yossi Gandelsman · Trevor Darrell · Amir Globerson · Alexei Efros

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #214
in Poster Session 3 »

How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification? Inspired by prompting in NLP, this paper investigates visual prompting: given input-output image example(s) of a new task at test time and a new input image, the goal is to automatically produce the output image, consistent with the given examples. We show that posing this problem as simple image inpainting -- literally just filling in a hole in a concatenated visual prompt image -- turns out to be surprisingly effective, provided that the inpainting algorithm has been trained on the right data. We train masked auto-encoders on a new dataset that we curated -- 88k unlabeled figures from academic papers sources on Arxiv. We apply visual prompting to these pretrained models and demonstrate results on various downstream image-to-image tasks, including foreground segmentation, single object detection, colorization, edge detection, etc. Project page: https://yossigandelsman.github.io/visual_prompt

Keywords: Computer Vision · Self-supervised learning · Visual Prompting
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Amir Bar (TAU / UC Berkeley)
Yossi Gandelsman (UC Berkeley)
Trevor Darrell (Electrical Engineering & Computer Science Department)
Amir Globerson (Tel Aviv University, Google)
Alexei Efros (UC Berkeley)

More from the Same Authors