Timezone: »

 
Benchmarking the Effect of Poisoning Defenses on the Security and Bias of the Final Model
Nathalie Baracaldo · Kevin Eykholt · Farhan Ahmed · Yi Zhou · Shriti Priya · Taesung Lee · Swanand Kadhe · Yusong Tan · Sridevi Polavaram · Sterling Suggs

@
in Trustworthy and Socially Responsible Machine Learning »
Event URL: https://openreview.net/forum?id=PP3H72O_E2f »

Machine learning models are susceptible to a class of attacks known as adversarial poisoning where an adversary can maliciously manipulate training data to hinder model performance or, more concerningly, insert backdoors to exploit at inference time. Many methods have been proposed to defend against adversarial poisoning by either identifying the poisoned samples to facilitate removal or developing poison agnostic training algorithms. Although effective, these proposed approaches can have unintended consequences on other aspects of model performance, such as worsening performance on certain data sub-populations, thus inducing a classification bias. In this work, we evaluate several adversarial poisoning defenses. In addition to traditional security metrics, i.e., robustness to poisoned samples, we propose a new metric to measure the potential undesirable discrimination of sub-populations resulting from using these defenses. Our investigation highlights that many of the evaluated defenses trade decision fairness to achieve higher adversarial poisoning robustness. Given these results, we recommend our proposed metric to be part of standard evaluations of machine learning defenses.

Author Information

Nathalie Baracaldo (IBM Research)

Nathalie Baracaldo leads the AI Security and Privacy Solutions team and is a Research Staff Member at IBM’s Almaden Research Center in San Jose, CA. Nathalie is passionate about delivering machine learning solutions that are highly accurate, withstand adversarial attacks and protect data privacy. Her team focuses on two main areas: federated learning, where models are trained without directly accessing training data and adversarial machine learning, where defenses are designed to withstand potential attacks to the machine learning pipeline. Nathalie is the primary investigator for the DARPA program Guaranteeing AI Robustness Against Deception (GARD), where AI security is investigated. Her team contributes to the Adversarial Robustness 360 Toolbox (ART). Nathalie is also the co-editor of the book: “Federated Learning: A Comprehensive Overview of Methods and Applications”, 2022 available in paper and as e-book in Springer, Apple books and Amazon. Nathalie's primary research interests lie at the intersection of information security, privacy and trust. As part of her work, she has also designed and implemented secure systems in the areas of cloud computing, Platform as a Service, secure data sharing and Internet of the Things. She has also contributed to projects to design scalable systems that monitor, manage performance and manage service level agreements in cloud environments. In 2020, Nathalie received the IBM Master Inventor distinction for her contributions to the IBM Intellectual Property and innovation. Nathalie also received the 2021 Corporate Technical Recognition, one of the highest recognitions provided to IBMers for breakthrough technical achievements that have led to notable market and industry success for IBM. This recognition was awarded for Nathalie's contribution to the Trusted AI initiative. Nathalie is associated Editor IEEE Transactions on Service Computing. Nathalie received her Ph.D. degree from the University of Pittsburgh in 2016. Her dissertation focused on preventing insider threats through the use of adaptive access control systems that integrate multiple sources of contextual information. Some of the topics that she has explored in the past include secure storage systems, privacy in online social networks, secure interoperability in distributed systems, risk management and trust evaluation. During her Ph.D. studies she received the 2014 Allen Kent Award for Outstanding Contributions to the Graduate Program in Information Science by the School of Information Sciences at the University of Pittsburgh. Nathalie also holds a master’s degree with Cum Laude distinction in computer sciences from the Universidad de los Andes, Colombia. Prior to that, she earned two undergraduate degrees in Computer Science and Industrial Engineering at the same university.

Kevin Eykholt (International Business Machines)
Farhan Ahmed (International Business Machines)
Yi Zhou (IBM Research)
Shriti Priya
Taesung Lee (IBM Research AI)
Swanand Kadhe (International Business Machines)
Yusong Tan (MITRE Corporation)
Sridevi Polavaram
Sterling Suggs (Two Six Technologies)

More from the Same Authors