Timezone: »
Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical in many real-world, high-stake applications.Hundreds of papers have either proposed new feature attribution methods, discussed or harnessed these tools in their work.However, despite humans being the target end-users, most attribution methods were only evaluated on proxy automatic-evaluation metrics (Zhang et al. 2018; Zhou et al. 2016; Petsiuk et al. 2018). In this paper, we conduct the first user study to measure attribution map effectiveness in assisting humans in ImageNet classification and Stanford Dogs fine-grained classification, and when an image is natural or adversarial (i.e., contains adversarial perturbations). Overall, feature attribution is surprisingly not more effective than showing humans nearest training-set examples. On a harder task of fine-grained dog categorization, presenting attribution maps to humans does not help, but instead hurts the performance of human-AI teams compared to AI alone. Importantly, we found automatic attribution-map evaluation measures to correlate poorly with the actual human-AI team performance. Our findings encourage the community to rigorously test their methods on the downstream human-in-the-loop applications and to rethink the existing evaluation metrics.
Author Information
Giang Nguyen (KAIST, South Korea)
Daeyoung Kim (KAIST)
Anh Nguyen (Auburn University)
More from the Same Authors
-
2021 : Nearest-neighbor is more useful than feature attribution in improving human accuracy on image classification »
Giang Nguyen · Anh Nguyen -
2023 Poster: ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification »
Mohammad Reza Taesiri · Giang Nguyen · Sarra Habchi · Cor-Paul Bezemer · Anh Nguyen -
2022 Poster: Visual correspondence-based explanations improve AI robustness and human-AI team accuracy »
Mohammad Reza Taesiri · Giang Nguyen · Anh Nguyen -
2017 : Invited Talk 4 »
Anh Nguyen -
2016 Demonstration: Adventures with Deep Generator Networks »
Jason Yosinski · Anh Nguyen · Jeff Clune · Douglas K Bemis -
2016 Poster: Synthesizing the preferred inputs for neurons in neural networks via deep generator networks »
Anh Nguyen · Alexey Dosovitskiy · Jason Yosinski · Thomas Brox · Jeff Clune