Timezone: »

 
Poster
SEGA: Instructing Text-to-Image Models using Semantic Guidance
Manuel Brack · Felix Friedrich · Dominik Hintersdorf · Lukas Struppek · Patrick Schramowski · Kristian Kersting

Wed Dec 13 08:45 AM -- 10:45 AM (PST) @ Great Hall & Hall B1+B2 #524
in Poster Session 3 »

Text-to-image diffusion models have recently received a lot of interest for their astonishing ability to produce high-fidelity images from text only. However, achieving one-shot generation that aligns with the user’s intent is nearly impossible, yet small changes to the input prompt often result in very different images. This leaves the user with little semantic control. To put the user in control, we show how to interact with the diffusion process to flexibly steer it along semantic directions. This semantic guidance (SEGA) generalizes to any generative architecture using classifier-free guidance. More importantly, it allows for subtle and extensive edits, composition and style changes, and optimizing the overall artistic conception. We demonstrate SEGA’s effectiveness on both latent and pixel-based diffusion models such as Stable Diffusion, Paella, and DeepFloyd-IF using a variety of tasks, thus providing strong evidence for its versatility and flexibility.

Author Information

Manuel Brack (DFKI)
Felix Friedrich (TU Darmstadt, hessian.AI)
Dominik Hintersdorf (TU Darmstadt)
Lukas Struppek (Technical University of Darmstadt)
Patrick Schramowski (DFKI, Hessian.AI, TU Darmstadt)
Kristian Kersting (TU Darmstadt)

More from the Same Authors