Skip to yearly menu bar Skip to main content

Workshop

Red Teaming GenAI: What Can We Learn from Adversaries?

Valeriia Cherepanova · Bo Li · Niv Cohen · Yifei Wang · Yisen Wang · Avital Shafran · Nil-Jana Akpinar · James Zou

[ OpenReview]

Abstract

The development and proliferation of modern generative AI models has introduced valuable capabilities, but these models and their applications also introduce risks to human safety. How do we identify risks in new systems before they cause harm during deployment? This workshop focuses on red teaming, an emergent adversarial approach to probing model behaviors, and its applications towards making modern generative AI safe for humans.

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles

9:00 AM

Coffee break

9:30 AM

Opening Remark

9:35 AM

Invited talk 1: Andy Zou and Q&A Invited Talk

Andy Zou

Video

10:10 AM

Invited talk 2: Danqi Chen on Uncovering Simple Failures in Generative Models and How to Fix Them Invited Talk

Danqi Chen

Video

10:45 AM

Contributed Talk 1: iART - Imitation guided Automated Red Teaming Oral

Sajad Mousavi · Desik Rengarajan · Ashwin Ramesh Babu · Vineet Gundecha · Avisek Naug · Sahand Ghorbanpour · Ricardo Luna Gutierrez · Antonio Guillen-Perez · Paolo Faraboschi · Soumyendu Sarkar

Video

10:55 AM

Contributed Talk 2: Failures to Find Transferable Image Jailbreaks Between Vision-Language Models Oral

Rylan Schaeffer · Dan Valentine · Luke Bailey · James Chua · Zane Durante · Cristobal Eyzaguirre · Joe Benton · Brando Miranda · Henry Sleight · Tony Wang · John Hughes · Rajashree Agrawal · Mrinank Sharma · Scott Emmons · Sanmi Koyejo · Ethan Perez

Video

11:05 AM

Contributed Talk 3: LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet Oral

Nathaniel Li · Ziwen Han · Ian Steneker · Willow Primack · Riley Goodside · Hugh Zhang · Zifan Wang · Cristina Menghini · Summer Yue

Video

11:20 AM

Panel Discussion Panel

Roei Schuster · Yaron Singer · Alex Tamkin · Bo Li

Video

12:00 PM

Lunch Lunch

12:00 PM

Poster Session Poster Session

1:50 PM

Invited talk 3: Niloofar Mireshghallah on A False Sense of Privacy: Semantic Leakage and Non-literal Copying in LLMs Invited Talk

Niloofar Mireshghallah

Video

2:15 PM

Invited talk 4: Jonas Geiping on When do adversarial attacks against language models matter? Invited Talk

Jonas Geiping

Video

3:00 PM

Coffee Break

3:30 PM

Invited talk 5: Vitaly Shmatikov and Q&A Invited Talk

Vitaly Shmatikov

Video

4:15 PM

Invited talk 6: Gowthami Somepalli and Q&A Invited Talk

Gowthami Somepalli

Video

4:30 PM

Contributed Talk 4: Rethinking LLM Memorization through the Lens of Adversarial Compression Oral

Avi Schwarzschild · Zhili Feng · Pratyush Maini · Zachary Lipton · J. Zico Kolter

Video

4:40 PM

Contributed Talk 5: A Realistic Threat Model for Large Language Model Jailbreaks Oral

Valentyn Boreiko · Alexander Panfilov · Vaclav Voracek · Matthias Hein · Jonas Geiping

Video

4:50 PM

Contributed Talk 6: Infecting LLM Agents via Generalizable Adversarial Attack Oral

Weichen Yu · Kai Hu · Tianyu Pang · Chao Du · Min Lin · Matt Fredrikson

Video

5:00 PM

Invited Talk 7: Max Kaufmann on Red-teaming AI systems in government Invited Talk

Max Kaufmann

Video

5:20 PM

Closing Remarks

Video

Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI Poster

Ambrish Rawat · Stefan Schoepf · Giulio Zizzo · Giandomenico Cornacchia · Muhammad Zaid Hameed · Kieran Fraser · Erik Miehling · Beat Buesser · Elizabeth Daly · Mark Purcell · Prasanna Sattigeri · Pin-Yu Chen · Kush Varshney

Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning Poster

Alex Beutel · Kai Xiao · Johannes Heidecke · Lilian Weng

MedAIScout: Automated Retrieval of Known Machine Learning Vulnerabilities in Medical Applications Poster

Athish Pranav Dharmalingam · Gargi Mitra

Infecting LLM Agents via Generalizable Adversarial Attack Poster

Weichen Yu · Kai Hu · Tianyu Pang · Chao Du · Min Lin · Matt Fredrikson

Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints Poster

Jonathan Noether · Adish Singla · Goran Radanovic

Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System Poster

Julian Collado · Kevin Stangl

Decoding Biases: An Analysis of Automated Methods and Metrics for Gender Bias Detection in Language Models Poster

Shachi H. Kumar · Saurav Sahay · Sahisnu Mazumder · Eda Okur · Ramesh Manuvinakurike · Nicole Beckage · Hsuan Su · Hung-yi Lee · Lama Nachman

Interactive Semantic Interventions for VLMs: Breaking VLMs with Human Ingenuity Poster

Lukas Klein · Kenza Amara · Carsten Lüth · Hendrik Strobelt · Mennatallah El-Assady · Paul Jaeger

Semantic Membership Inference Attack against Large Language Models Poster

Hamid Mozaffari · Virendra Marathe

Rethinking LLM Memorization through the Lens of Adversarial Compression Poster

Avi Schwarzschild · Zhili Feng · Pratyush Maini · Zachary Lipton · J. Zico Kolter

Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning Poster

Seanie Lee · Minsu Kim · Lynn Cherif · David Dobre · Juho Lee · Sung Ju Hwang · Kenji Kawaguchi · Gauthier Gidel · Yoshua Bengio · Nikolay Malkin · Moksh Jain

Between the Bars: Gradient-based Jailbreaks are Bugs that induce Features Poster

Kaivalya Hariharan · Uzay Girit

Large Language Model Detoxification: Data and Metric Solutions Poster

SungJoo Byun · HYOPIL SHIN

SAGE-RT: Synthetic Alignment data Generation for Safety Evaluation and Red Teaming Poster

Anurakt Kumar · Divyanshu Kumar · Jatan Loya · Nitin Aravind Birur · Tanay Baswa · Sahil Agarwal · Prashanth Harshangi

An Adversarial Perspective on Machine Unlearning for AI Safety Poster

Jakub Łucki · Boyi Wei · Yangsibo Huang · Peter Henderson · Florian Tramer · Javier Rando

Stability Evaluation of Large Language Models via Distributional Perturbation Analysis Poster

Jiashuo Liu · Jiajin Li · Peng Cui · Jose Blanchet

Lessons From Red Teaming 100 Generative AI Products Poster

Blake Bullwinkel · Amanda Minnich · Shiven Chawla · Gary Lopez Munoz · Martin Pouliot · Whitney Maxwell · Joris de Gruyter · Katherine Pratt · Saphir Qi · Nina Chikanov · Roman Lutz · Raja Sekhar Rao Dheekonda · Bolor-Erdene Jagdagdorj · Rich Lundeen · Sam Vaughan · Victoria Westerhoff · Pete Bryan · Ram Shankar Siva Kumar · Yonatan Zunger · Mark Russinovich

LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet Poster

Nathaniel Li · Ziwen Han · Ian Steneker · Willow Primack · Riley Goodside · Hugh Zhang · Zifan Wang · Cristina Menghini · Summer Yue

Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage Poster

Rafi Rashid · Jing Liu · Toshiaki Koike-Akino · Shagufta Mehnaz · Ye Wang

Steganography in Large Language Models: Investigating Emergence and Mitigations Poster

Yohan Mathew · Robert McCarthy · Ollie Matthews · Joan Velja · Nandi Schoots · Dylan Cope

A Realistic Threat Model for Large Language Model Jailbreaks Poster

Valentyn Boreiko · Alexander Panfilov · Vaclav Voracek · Matthias Hein · Jonas Geiping

Decompose, Recompose, and Conquer: Multi-modal LLMs are Vulnerable to Compositional Adversarial Attacks in Multi-Image Queries Poster

Julius Broomfield · George Ingebretsen · Reihaneh Iranmanesh · Sara Pieri · Ethan Kosak-Hine · Tom Gibbs · Reihaneh Rabbany · Kellin Pelrine

Failures to Find Transferable Image Jailbreaks Between Vision-Language Models Poster

Rylan Schaeffer · Dan Valentine · Luke Bailey · James Chua · Zane Durante · Cristobal Eyzaguirre · Joe Benton · Brando Miranda · Henry Sleight · Tony Wang · John Hughes · Rajashree Agrawal · Mrinank Sharma · Scott Emmons · Sanmi Koyejo · Ethan Perez

SkewAct: Red Teaming Large Language Models via Activation-Skewed Adversarial Prompt Optimization Poster

Hanxi Guo · Siyuan Cheng · Guanhong Tao · Guangyu Shen · Zhuo Zhang · Shengwei An · Kaiyuan Zhang · Xiangyu Zhang

Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs Poster

Aly Kassem · Omar Mahmoud · Niloofar Mireshghallah · Hyunwoo Kim · Yulia Tsvetkov · Yejin Choi · Sherif Saad · Santu Rana

TOFU: A Task of Fictitious Unlearning for LLMs Poster

Pratyush Maini · Zhili Feng · Avi Schwarzschild · Zachary Lipton · J. Zico Kolter

iART - Imitation guided Automated Red Teaming Poster

Sajad Mousavi · Desik Rengarajan · Ashwin Ramesh Babu · Vineet Gundecha · Avisek Naug · Sahand Ghorbanpour · Ricardo Luna Gutierrez · Antonio Guillen-Perez · Paolo Faraboschi · Soumyendu Sarkar

Does Refusal Training in LLMs Generalize to the Past Tense? Poster

Maksym Andriushchenko · Nicolas Flammarion

Plentiful Jailbreaks with String Compositions Poster

Brian Huang

Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models Poster

Hongfu Liu · Yuxi Xie · Ye Wang · Michael Qizhe Shieh

CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation Poster

Tong Chen · Akari Asai · Niloofar Mireshghallah · Sewon Min · James Grimmelmann · Yejin Choi · Hannaneh Hajishirzi · Luke Zettlemoyer · Pang Wei Koh

Curiosity-driven Red teaming for Large Language Models Poster

Zhang-Wei Hong · Idan Shenfeld · Tsun-Hsuan Johnson Wang · Yung-Sung Chuang · Aldo Pareja · Jim Glass · Akash Srivastava · Pulkit Agrawal

Adversarial Negotiation Dynamics in Generative Language Models Poster

Arinbjörn Kolbeinsson · Benedikt Kolbeinsson

LLM-Assisted Red Teaming of Diffusion Models through "Failures Are Fated, But Can Be Faded" Poster

Som Sagar · Aditya Taparia · Ransalu Senanayake

Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding Poster

Haneul Yoo · Yongjin Yang · Hwaran Lee

What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks Poster

Nathalie Kirch · Severin Field · Stephen Casper

Algorithmic Oversight for Deceptive Reasoning Poster

Ege Onur Taga · Mingchen Li · Yongqi Chen · Samet Oymak

A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation Poster

aviral srivastava · Sourav Panda