Timezone: »
Following growing concerns with both harmful research impact and research conduct in computer science, including concerns with research published at NeurIPS, this year’s conference introduced two new mechanisms for ethical oversight: a requirement that authors include a “broader impact statement” in their paper submissions and additional evaluation criteria asking paper reviewers to identify any potential ethical issues with the submissions.
These efforts reflect a recognition that existing research norms have failed to address the impacts of AI research, and take place against the backdrop of a larger reckoning with the role of AI in perpetuating injustice. The changes have been met with both praise and criticism some within and outside the community see them as a crucial first step towards integrating ethical reflection and review into the research process, fostering necessary changes to protect populations at risk of harm. Others worry that AI researchers are not well placed to recognize and reason about the potential impacts of their work, as effective ethical deliberation may require different expertise and the involvement of other stakeholders.
This debate reveals that even as the AI research community is beginning to grapple with the legitimacy of certain research questions and critically reflect on its research practices, there remains many open questions about how to ensure effective ethical oversight. This workshop therefore aims to examine how concerns with harmful impacts should affect the way the research community develops its research agendas, conducts its research, evaluates its research contributions, and handles the publication and dissemination of its findings. This event complements other NeurIPS workshops this year devoted to normative issues in AI and builds on others from years past, but adopts a distinct focus on the ethics of research practice and the ethical obligations of researchers.
Sat 5:30 a.m. - 5:45 a.m.
|
Welcome
|
🔗 |
Sat 5:45 a.m. - 6:15 a.m.
|
Morning keynote
(
Keynote
)
|
Hanna Wallach · Rosie Campbell 🔗 |
Sat 6:15 a.m. - 7:15 a.m.
|
Ethical oversight in the peer review process
(
Discussion panel
)
Discussion on potential reflection and oversight interventions such as 'broader impact statements' and their effectiveness |
Sarah Brown · Heather Douglas · Iason Gabriel · Brent Hecht · Rosie Campbell 🔗 |
Sat 7:15 a.m. - 7:30 a.m.
|
Morning break
|
🔗 |
Sat 7:30 a.m. - 8:30 a.m.
|
Harms from AI research
(
Discussion panel
)
Case studies and mitigation approaches |
Anna Lauren Hoffmann · Nyalleng Moorosi · Vinay Prabhu · Deborah Raji · Jacob Metcalf · Sherry Stanley 🔗 |
Sat 8:30 a.m. - 9:30 a.m.
|
How should researchers engage with controversial applications of AI?
(
Discussion panel
)
|
Logan Koepke · CATHERINE ONEIL · Tawana Petty · Cynthia Rudin · Deborah Raji · Shawn Bushway 🔗 |
Sat 9:30 a.m. - 10:30 a.m.
|
Lunch and watch lightning talks (in parallel) from workshop submissions
link »
In advance of the next session (which is discussions with paper authors), please take some time to watch the videos of the submitted papers over lunch. |
🔗 |
Sat 10:30 a.m. - 11:30 a.m.
|
Discussions with authors of submitted papers
(
Breakouts
)
link »
Please join our Gather Town to meet the paper authors! - Walk up to the title of the paper you're interested in, then press 'x' to view the paper - There is a lounge area to the North of the paper discussion room if you would like to have more informal conversations |
🔗 |
Sat 11:30 a.m. - 12:30 p.m.
|
Responsible publication: NLP case study
(
Discussion panel
)
|
Miles Brundage · Bryan McCann · Colin Raffel · Natalie Schulter · Zeerak Waseem · Rosie Campbell 🔗 |
Sat 12:30 p.m. - 12:45 p.m.
|
Afternoon break
|
🔗 |
Sat 12:45 p.m. - 1:45 p.m.
|
Strategies for anticipating and mitigating risks
(
Discussion panel
)
|
Ashley Casovan · Timnit Gebru · Shakir Mohamed · Solon Barocas · Aviv Ovadya 🔗 |
Sat 1:45 p.m. - 2:45 p.m.
|
The roles of different parts of the research ecosystem in navigating broader impacts
(
Discussion panel
)
|
Josh Greenberg · Liesbeth Venema · Ben Zevenbergen · Lilly Irani · Solon Barocas 🔗 |
Sat 2:45 p.m. - 3:00 p.m.
|
Closing remarks
|
🔗 |
-
|
Auditing Government AI: Assessing ethical vulnerability of machine learning
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Alayna A Kennedy 🔗 |
-
|
An Ethical Highlighter for People-Centric Dataset Creation
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Margot Hanley · Apoorv Khandelwal · Hadar Averbuch-Elor · Noah Snavely · Helen Nissenbaum 🔗 |
-
|
The Managerial Effects of Algorithmic Fairness Activism
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Bo Cowgill · Fabrizio Dell'Acqua · Sandra Matz 🔗 |
-
|
Biased Programmers? Or Biased Data? A Field Experiment in Operationalizing AI Ethics
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Bo Cowgill · Fabrizio Dell'Acqua · Augustin Chaintreau · Nakul Verma · Samuel Deng · Daniel Hsu 🔗 |
-
|
Ethical Testing in the Real World: Recommendations for Physical Testing of Adversarial Machine Learning Attacks
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Ram Shankar Siva Kumar · Maggie Delano · Kendra Albert · Afsaneh Rigot · Jonathon Penney 🔗 |
-
|
Nose to Glass: Looking In to Get Beyond
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Josephine Seah 🔗 |
-
|
Training Ethically Responsible AI Researchers: a Case Study
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Hang Yuan · Claudia Vanea · Federica Lucivero · Nina Hallowell 🔗 |
-
|
Like a Researcher Stating Broader Impact For the Very First Time
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Grace Abuhamad · Claudel Rheault 🔗 |
-
|
Anticipatory Ethics and the Role of Uncertainty
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Priyanka Nanayakkara · Nicholas Diakopoulos · Jessica Hullman 🔗 |
-
|
Non-Portability of Algorithmic Fairness in India
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Nithya Sambasivan · Erin Arnesen · Ben Hutchinson · Vinodkumar Prabhakaran 🔗 |
-
|
An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
David Tran · Alex Valtchanov · Keshav R Ganapathy · Raymond Feng · Eric Slud · Micah Goldblum · Tom Goldstein 🔗 |
-
|
AI in the “Real World”: Examining the Impact of AI Deployment in Low-Resource Contexts
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Chinasa T. Okolo 🔗 |
-
|
Ideal theory in AI ethics
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Daniel Estrada 🔗 |
-
|
Overcoming Failures of Imagination in AI Infused System Development and Deployment
(
Lightning talk (5-7 mins)
)
SlidesLive Video » |
Margarita Boyarskaya · Alexandra Olteanu · Kate Crawford 🔗 |
Author Information
Carolyn Ashurst (FHI, University of Oxford)
Rosie Campbell (Partnership on AI)
Deborah Raji (Mozilla Foundation)
Solon Barocas
Stuart Russell (UC Berkeley)
More from the Same Authors
-
2021 Spotlight: Uncertain Decisions Facilitate Better Preference Learning »
Cassidy Laidlaw · Stuart Russell -
2021 : An Empirical Investigation of Representation Learning for Imitation »
Cynthia Chen · Sam Toyer · Cody Wild · Scott Emmons · Ian Fischer · Kuang-Huei Lee · Neel Alex · Steven Wang · Ping Luo · Stuart Russell · Pieter Abbeel · Rohin Shah -
2021 : AI and the Everything in the Whole Wide World Benchmark »
Deborah Raji · Emily Denton · Emily M. Bender · Alex Hanna · Amandalynne Paullada -
2021 : Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning »
Thomas Liao · Rohan Taori · Deborah Raji · Ludwig Schmidt -
2021 : Cross-Domain Imitation Learning via Optimal Transport »
Arnaud Fickinger · Samuel Cohen · Stuart Russell · Brandon Amos -
2022 : Adversarial Policies Beat Professional-Level Go AIs »
Tony Wang · Adam Gleave · Nora Belrose · Tom Tseng · Michael Dennis · Yawen Duan · Viktor Pogrebniak · Joseph Miller · Sergey Levine · Stuart Russell -
2022 Social: Ethics Review - Open Discussion »
Deborah Raji · William Isaac · Cherie Poland · Alexandra Luccioni -
2021 : Evaluation as a Process for Engineering Responsibility in AI »
Deborah Raji -
2021 : BASALT: A MineRL Competition on Solving Human-Judged Task + Q&A »
Rohin Shah · Cody Wild · Steven Wang · Neel Alex · Brandon Houghton · William Guss · Sharada Mohanty · Stephanie Milani · Nicholay Topin · Pieter Abbeel · Stuart Russell · Anca Dragan -
2021 Poster: Scalable Online Planning via Reinforcement Learning Fine-Tuning »
Arnaud Fickinger · Hengyuan Hu · Brandon Amos · Stuart Russell · Noam Brown -
2021 Poster: Uncertain Decisions Facilitate Better Preference Learning »
Cassidy Laidlaw · Stuart Russell -
2021 Poster: Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism »
Paria Rashidinejad · Banghua Zhu · Cong Ma · Jiantao Jiao · Stuart Russell -
2021 Poster: MADE: Exploration via Maximizing Deviation from Explored Regions »
Tianjun Zhang · Paria Rashidinejad · Jiantao Jiao · Yuandong Tian · Joseph Gonzalez · Stuart Russell -
2020 : How should researchers engage with controversial applications of AI? »
Logan Koepke · CATHERINE ONEIL · Tawana Petty · Cynthia Rudin · Deborah Raji · Shawn Bushway -
2020 : Harms from AI research »
Anna Lauren Hoffmann · Nyalleng Moorosi · Vinay Prabhu · Deborah Raji · Jacob Metcalf · Sherry Stanley -
2020 : Morning keynote »
Hanna Wallach · Rosie Campbell -
2020 : AI and the Everything in the Whole Wide World Benchmark »
Deborah Raji -
2020 : Invited Talk 3: Inioluwa Deborah Raji »
Deborah Raji -
2020 : Panel »
Kilian Weinberger · Maria De-Arteaga · Shibani Santurkar · Jonathan Frankle · Deborah Raji -
2020 Poster: The MAGICAL Benchmark for Robust Imitation »
Sam Toyer · Rohin Shah · Andrew Critch · Stuart Russell -
2020 Poster: SLIP: Learning to Predict in Unknown Dynamical Systems with Long-Term Memory »
Paria Rashidinejad · Jiantao Jiao · Stuart Russell -
2020 Oral: SLIP: Learning to Predict in Unknown Dynamical Systems with Long-Term Memory »
Paria Rashidinejad · Jiantao Jiao · Stuart Russell -
2020 Poster: Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design »
Michael Dennis · Natasha Jaques · Eugene Vinitsky · Alexandre Bayen · Stuart Russell · Andrew Critch · Sergey Levine -
2020 Oral: Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design »
Michael Dennis · Natasha Jaques · Eugene Vinitsky · Alexandre Bayen · Stuart Russell · Andrew Critch · Sergey Levine -
2019 : AI's Blindspots and Where to Find Them »
Deborah Raji -
2018 Poster: Meta-Learning MCMC Proposals »
Tongzhou Wang · YI WU · Dave Moore · Stuart Russell -
2018 Poster: Learning Plannable Representations with Causal InfoGAN »
Thanard Kurutach · Aviv Tamar · Ge Yang · Stuart Russell · Pieter Abbeel