Timezone: »

 
Poster
Visual Prompting via Image Inpainting
Amir Bar · Yossi Gandelsman · Trevor Darrell · Amir Globerson · Alexei Efros

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #214
in Poster Session 3 »

How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification? Inspired by prompting in NLP, this paper investigates visual prompting: given input-output image example(s) of a new task at test time and a new input image, the goal is to automatically produce the output image, consistent with the given examples. We show that posing this problem as simple image inpainting -- literally just filling in a hole in a concatenated visual prompt image -- turns out to be surprisingly effective, provided that the inpainting algorithm has been trained on the right data. We train masked auto-encoders on a new dataset that we curated -- 88k unlabeled figures from academic papers sources on Arxiv. We apply visual prompting to these pretrained models and demonstrate results on various downstream image-to-image tasks, including foreground segmentation, single object detection, colorization, edge detection, etc. Project page: https://yossigandelsman.github.io/visual_prompt

Keywords: Computer Vision · Self-supervised learning · Visual Prompting
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Amir Bar (TAU / UC Berkeley)
Amir Bar

Amir Bar is a fourth-year Ph.D. candidate at Tel Aviv University and a Visiting Ph.D. Researcher at UC Berkeley, advised by Amir Globerson and Trevor Darrell. His primary research area centers around self-supervised learning and how to use large amounts of unlabeled images and videos to enable computers to develop visual understanding. Lately, his focus has been on improving learning algorithms for Masked Image Modeling and Visual Prompting, which involves adapting computer vision models during test time for novel computer vision tasks without changing the model weights or task-specific fine-tuning.

Yossi Gandelsman (UC Berkeley)
Trevor Darrell (Electrical Engineering & Computer Science Department)
Amir Globerson (Tel Aviv University, Google)
Alexei Efros (UC Berkeley)

More from the Same Authors