Timezone: »
Counterfactual explanations provide means for prescriptive model explanations by suggesting actionable feature changes (e.g., increase income) that allow individuals to achieve favourable outcomes in the future (e.g., insurance approval).Choosing an appropriate method is a crucial aspect for meaningful counterfactual explanations. As documented in recent reviews, there exists a quickly growing literature with available methods. Yet, in the absence of widely available open--source implementations, the decision in favour of certain models is primarily based on what is readily available. Going forward -- to guarantee meaningful comparisons across explanation methods -- we present \texttt{CARLA} (\textbf{C}ounterfactual \textbf{A}nd \textbf{R}ecourse \textbf{L}ibr\textbf{A}ry), a python library for benchmarking counterfactual explanation methods across both different data sets and different machine learning models. In summary, our work provides the following contributions: (i) an extensive benchmark of 11 popular counterfactual explanation methods, (ii) a benchmarking framework for research on future counterfactual explanation methods, and (iii) a standardized set of integrated evaluation measures and data sets for transparent and extensive comparisons of these methods.We have open sourced \texttt{CARLA} and our experimental results on \href{https://github.com/indyfree/CARLA}{Github}, making them available as competitive baselines. We welcome contributions from other research groups and practitioners.
Author Information
Martin Pawelczyk (University of Tübingen)
# Academic Exp ## Phd Student at Uni of Tübingen, Germany: ## MSc Statistics, London School of Economics, UK ## MSc Econometrics, University of Edinburgh, UK ## BSc Economics, University of Cologne, Germany # Work Exp ## ML intern at SDG financing Lab, OECD, Paris ## Working student at r2b energy consulting, Cologne
Sascha Bielawski (University of Tuebingen)
Johan Van den Heuvel (University of Tuebingen)
Tobias Richter (University of Cologne)
Gjergji Kasneci (University of Tuebingen)
More from the Same Authors
-
2021 : A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines »
Vadim Borisov · Johannes Meier · Johan Van den Heuvel · Hamed Jalali · Gjergji Kasneci -
2021 : Gaussian Graphical Models as an Ensemble Method for Distributed Gaussian Processes »
Hamed Jalali · Gjergji Kasneci -
2022 : Expert Selection in Distributed Gaussian Processes: A Multi-label Classification Approach »
Hamed Jalali · Gjergji Kasneci -
2022 : I Prefer not to Say – Operationalizing Fair and User-guided Data Minimization »
Tobias Leemann · Martin Pawelczyk · Christian Eberle · Gjergji Kasneci -
2022 : Explanation Shift: Detecting distribution shifts on tabular data via the explanation space »
Carlos Mougan · Klaus Broelemann · Gjergji Kasneci · Thanassis Tiropanis · Steffen Staab -
2022 : On the Trade-Off between Actionable Explanations and the Right to be Forgotten »
Martin Pawelczyk · Tobias Leemann · Asia Biega · Gjergji Kasneci -
2022 Poster: OpenXAI: Towards a Transparent Evaluation of Model Explanations »
Chirag Agarwal · Satyapriya Krishna · Eshika Saxena · Martin Pawelczyk · Nari Johnson · Isha Puri · Marinka Zitnik · Himabindu Lakkaraju -
2021 : [S4] A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines »
Vadim Borisov · Johannes Meier · Johan Van den Heuvel · Hamed Jalali · Gjergji Kasneci -
2019 : Poster session »
Jindong Gu · Alice Xiang · Atoosa Kasirzadeh · Zhiwei Han · Omar U. Florez · Frederik Harder · An-phi Nguyen · Amir Hossein Akhavan Rahnama · Michele Donini · Dylan Slack · Junaid Ali · Paramita Koley · Michiel Bakker · Anna Hilgard · Hailey James · Gonzalo Ramos · Jialin Lu · Jingying Yang · Margarita Boyarskaya · Martin Pawelczyk · Kacper Sokol · Mimansa Jaiswal · Umang Bhatt · David Alvarez-Melis · Aditya Grover · Charles Marx · Mengjiao (Sherry) Yang · Jingyan Wang · Gökhan Çapan · Hanchen Wang · Steffen Grünewälder · Moein Khajehnejad · Gourab Patro · Russell Kunes · Samuel Deng · Yuanting Liu · Luca Oneto · Mengze Li · Thomas Weber · Stefan Matthes · Duy Patrick Tu