Timezone: »

 
Poster
Diffusion Models as Plug-and-Play Priors
Alexandros Graikos · Nikolay Malkin · Nebojsa Jojic · Dimitris Samaras

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #528
in Poster Session 4 »
We consider the problem of inferring high-dimensional data $x$ in a model that consists of a prior $p(x)$ and an auxiliary differentiable constraint $c(x,y)$ on $x$ given some additional information $y$. In this paper, the prior is an independently trained denoising diffusion generative model. The auxiliary constraint is expected to have a differentiable form, but can come from diverse sources. The possibility of such inference turns diffusion models into plug-and-play modules, thereby allowing a range of potential applications in adapting models to new domains and tasks, such as conditional generation or image segmentation. The structure of diffusion models allows us to perform approximate inference by iterating differentiation through the fixed denoising network enriched with different amounts of noise at each step. Considering many noised versions of $x$ in evaluation of its fitness is a novel search mechanism that may lead to new algorithms for solving combinatorial optimization problems. The code is available at https://github.com/AlexGraikos/diffusion_priors.
Keywords: Image Segmentation · diffusion models · conditional generation

Author Information

Alexandros Graikos (State University of New York, Stony Brook)
Alexandros Graikos

Ph.D. at Stony Brook university. Interested in ML and Computer vision.

Nikolay Malkin (Mila / Université de Montréal)
Nebojsa Jojic (Microsoft Research)
Dimitris Samaras (Stony Brook University)

More from the Same Authors