We propose a novel confidence metric, namely, attribution-based confidence (ABC) for deep neural networks (DNNs). ABC metric characterizes whether the output of a DNN on an input can be trusted. DNNs are known to be brittle on inputs outside the training distribution and are, hence, susceptible to adversarial attacks. This fragility is compounded by a lack of effectively computable measures of model confidence that correlate well with the accuracy of DNNs. These factors have impeded the adoption of DNNs in high-assurance systems. The proposed ABC metric addresses these challenges. It does not require access to the training data, the use of ensembles, or the need to train a calibration model on a held-out validation set. Hence, the new metric is usable even when only a trained model is available for inference. We mathematically motivate the proposed metric and evaluate its effectiveness with two sets of experiments. First, we study the change in accuracy and the associated confidence over out-of-distribution inputs. Second, we consider several digital and physically realizable attacks such as FGSM, CW, DeepFool, PGD, and adversarial patch generation methods. The ABC metric is low on out-of-distribution data and adversarial examples, where the accuracy of the model is also low. These experiments demonstrate the effectiveness of the ABC metric to make DNNs more trustworthy and resilient.
Susmit Jha (SRI)
Sunny Raj (University of Central Florida)
Steven Fernandes (University of Central Florida)
Sumit K Jha (University of Central Florida)
Dr. Sumit K. Jha is an Associate Professor of Computer Science at the University of Central Florida (UCF), Orlando. Dr. Jha joined the University of Central Florida in 2010 after receiving his Ph.D. in Computer Science at Carnegie Mellon University. Before joining Carnegie Mellon, he graduated with B.Tech (Honors) in Computer Science and Engineering from the Indian Institute of Technology Kharagpur in 2004. Dr. Jha has worked on R&D problems at Microsoft Research India, General Motors, INRIA France and the Air Force Research Lab Information Directorate. His research has been supported by the National Science Foundation, the Air Force Office of Scientific Research, the Oak Ridge National Laboratory, the Royal Bank of Canada, the Florida Center for Cybersecurity, and the Air Force Research Laboratory. He is a full member of the Sigma Xi and is a recipient of the IEEE Orlando Engineering Educator Excellence Award. Dr. Jha was awarded the prestigious Air Force Young Investigator Award in 2016 and his research has led to three Best Paper awards.
Somesh Jha (University of Wisconsin, Madison)
Brian Jalaian (U.S. Army Research Laboratory)
Gunjan Verma (U.S. Army Research Laboratory)
Ananthram Swami (Army Research Laboratory, Adelphi)
More from the Same Authors
2019 Poster: Robust Attribution Regularization »
Jiefeng Chen · Xi Wu · Vaibhav Rastogi · Yingyu Liang · Somesh Jha
2019 Poster: Error Correcting Output Codes Improve Probability Estimation and Adversarial Robustness of Deep Neural Networks »
Gunjan Verma · Ananthram Swami