Timezone: »
Generative models (e.g., GANs, diffusion models) learn the underlying data distribution in an unsupervised manner. However, many applications of interest require sampling from a particular region of the output space or sampling evenly over a range of characteristics. For efficient sampling in these scenarios, we propose Generative Visual Prompt (PromptGen), a framework for distributional control over pre-trained generative models by incorporating knowledge of other off-the-shelf models. PromptGen defines control as energy-based models (EBMs) and samples images in a feed-forward manner by approximating the EBM with invertible neural networks, avoiding optimization at inference. Our experiments demonstrate how PromptGen can efficiently sample from several unconditional generative models (e.g., StyleGAN2, StyleNeRF, diffusion autoencoder, NVAE) in a controlled or/and de-biased manner using various off-the-shelf models: (1) with the CLIP model as control, PromptGen can sample images guided by text, (2) with image classifiers as control, PromptGen can de-bias generative models across a set of attributes or attribute combinations, and (3) with inverse graphics models as control, PromptGen can sample images of the same identity in different poses. (4) Finally, PromptGen reveals that the CLIP model shows a "reporting bias" when used as control, and PromptGen can further de-bias this controlled distribution in an iterative manner. The code is available at https://github.com/ChenWu98/Generative-Visual-Prompt.
Author Information
Chen Henry Wu (Carnegie Mellon University)
I am working on generative models.
Saman Motamed (King Abdullah University of Science and Technology)
Shaunak Srivastava (CMU, Carnegie Mellon University)
Fernando D De la Torre (Carnegie Mellon University)
More from the Same Authors
-
2022 : Making Text-to-Image Diffusion Models Zero-Shot Image-to-Image Editors by Inferring "Random Seeds" »
Chen Henry Wu · Fernando D De la Torre -
2011 Poster: Matrix Completion for Image Classification »
Ricardo S Cabral · Fernando D De la Torre · Joao P Costeira · Alexandre Bernardino -
2009 Poster: Canonical Time Warping for Alignment of Human Behavior »
Feng Zhou · Fernando D De la Torre -
2008 Poster: Robust Kernel Principal Component Analysis »
Minh Hoai Nguyen · Fernando D De la Torre -
2008 Spotlight: Robust Kernel Principal Component Analysis »
Minh Hoai Nguyen · Fernando D De la Torre