Unexploited Information Value in Human-AI Collaboration

Humans and AIs are often paired on decision tasks with the expectation of achievingcomplementary performance -- where the combination of human and AI outper-forms either one alone. However, how to improve performance of a human-AIteam is often not clear without knowing more about what particular informationand strategies each agent employs. In this paper, we propose a model based instatistically decision theory to analyze human-AI collaboration from the perspec-tive of what information could be used to improve a human or AI decision. Wedemonstrate our model on a deepfake detection task to investigate seven video-levelfeatures by their unexploited value of information. We compare the human alone,AI alone and human-AI team and offer insights on how the AI assistance impactspeople’s usage of the information and what information that the AI exploits wellmight be useful for improving human decisions.