Timezone: »

 
Poster
On the Accuracy of Influence Functions for Measuring Group Effects
Pang Wei Koh · Kai-Siang Ang · Hubert Teo · Percy Liang

Thu Dec 12 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #82
in Applications -- Fairness, Accountability, and Transparency »

Influence functions estimate the effect of removing a training point on a model without the need to retrain. They are based on a first-order Taylor approximation that is guaranteed to be accurate for sufficiently small changes to the model, and so are commonly used to study the effect of individual points in large datasets. However, we often want to study the effects of large groups of training points, e.g., to diagnose batch effects or apportion credit between different data sources. Removing such large groups can result in significant changes to the model. Are influence functions still accurate in this setting? In this paper, we find that across many different types of groups and for a range of real-world datasets, the predicted effect (using influence functions) of a group correlates surprisingly well with its actual effect, even if the absolute and relative errors are large. Our theoretical analysis shows that such strong correlation arises only under certain settings and need not hold in general, indicating that real-world datasets have particular properties that allow the influence approximation to be accurate.

Keywords: Fairness, Accountability, and Transparency · Applications
[ Paper]  [ Poster] 

Author Information

Pang Wei Koh (Stanford University)
Kai-Siang Ang (Stanford University)
Hubert Teo (Stanford University)
Percy Liang (Stanford University)
Percy Liang

Percy Liang is an Assistant Professor of Computer Science at Stanford University (B.S. from MIT, 2004; Ph.D. from UC Berkeley, 2011). His research spans machine learning and natural language processing, with the goal of developing trustworthy agents that can communicate effectively with people and improve over time through interaction. Specific topics include question answering, dialogue, program induction, interactive learning, and reliable machine learning. His awards include the IJCAI Computers and Thought Award (2016), an NSF CAREER Award (2016), a Sloan Research Fellowship (2015), and a Microsoft Research Faculty Fellowship (2014).

More from the Same Authors