Timezone: »

Red-Teaming the Stable Diffusion Safety Filter
Javier Rando · Daniel Paleka · David Lindner · Lennart Heim · Florian Tramer

Stable Diffusion is a recent open-source image generation model comparable to proprietary models such as DALL·E, Imagen, or Parti. Stable Diffusion comes with a safety filter that aims to prevent generating explicit images. Unfortunately, the filter is obfuscated and poorly documented. This makes it hard for users to prevent misuse in their applications, and to understand the filter's limitations and improve it. We first show that it is easy to generate disturbing content that bypasses the safety filter. We then reverse-engineer the filter and find that while it aims to prevent sexual content, it ignores violence, gore, and other similarly disturbing content. Based on our analysis, we argue safety measures in future model releases should strive to be fully open and properly documented to stimulate security contributions from the community.

Author Information

Javier Rando (ETH Zürich)
Daniel Paleka (ETH Zurich)
David Lindner (ETH Zurich)
Lennart Heim (Centre for Governance of AI)
Lennart Heim

Lennart Heim is an AI Governance researcher at the Centre for the Governance of AI in Oxford. He focuses on compute governance. His research interests include the role of compute in the AI production function, the compute supply chain, forecasting emerging technologies, and the security of AI systems. He’s also a member of the OECD.AI Expert Group on AI Compute and Climate. He has a background in Computer Engineering.

Florian Tramer (ETH Zürich)

More from the Same Authors