Timezone: »
Even with the rise of unsupervised learning and weak supervision techniques, human-labeled data is still a necessary part of machine learning pipelines in many real-world contexts and applications. This often involves using crowdworkers for the laborious task of labeling large amounts of data. This is a largely asynchronous process and can lead to conflict among the workers, where individual labelers potentially submit labels in disagreement from each other for a given data item. When such noisy data is fed to a machine learning model, the accuracy and performance (on test data) of the overall system can suffer. One popular workaround is to entirely discard the data items with conflict. This however, leads to wastage of expensive, human-supplied data. Moreover, the data points with conflicting labels often are the data points which are crucial in determining the decision boundaries for the model itself. Another possibility is to automate conflict resolution. Here however, given humans themselves are in disagreement, state-of-the-art models can not be expected to reliably solve the problem. In practice therefore, it becomes imperative for a human to step in and resolve the conflict. Given conflict resolution is a non-trivial task, assistance of expensive subject matter experts (SMEs) is required. To help manage the SME’s time more efficiently, we propose an intelligent approach to resolve label conflicts by automatically re-ranking the conflicts in such an order that the conflicts with the most missing information useful to the model are displayed first, complete with ML assistance to auto-resolve easy conflicts, and explanations for justifying decisions and improving explainability.
Author Information
Narendra Nath Joshi (IBM Research)
Aabhas Sharma (IBM Research)
Michelle Brachman (IBM Research AI)
Qian Pan (IBM Research AI)
Michael Muller (IBM Research)
Michael Muller works in the AI Interactions group of IBM Research AI, where his work focuses on the human aspects of data science; ethics and values in applications of AI to human issues; metrics and analytics for enterprise social software applications, with particular application to employee engagement emergent social phenomena in social software. Recognitions include: ACM Distinguished Scientist; SIGCHI Academy; IBM Master Inventor. Steering Committees: EUSSET (European Society for the study of Socially Embedded Technologies); ACM GROUP conference series. Papers co-chair for ECSCW 2019 (European Computer Supported Cooperative Work conference).
Michael Desmond (IBM Research)
Kristina Brimijoin (IBM Research)
Zahra Ashktorab (IBM Research AI)
Evelyn Duesterwald (IBM Research)
Casey Dugan (IBM Research AI)
More from the Same Authors
-
2022 Workshop: HCAI@NeurIPS 2022, Human Centered AI »
Michael Muller · Plamen P Angelov · Hal Daumé III · Shion Guha · Q.Vera Liao · Nuria Oliver · David Piorkowski -
2021 Workshop: Human Centered AI »
Michael Muller · Plamen P Angelov · Shion Guha · Marina Kogan · Gina Neff · Nuria Oliver · Manuel Rodriguez · Adrian Weller -
2020 Demonstration: AI Assisted Data Labeling »
Michael Desmond · Evelyn Duesterwald · Kristina Brimijoin · Michael Muller · Aabhas Sharma · Narendra Nath Joshi · Qian Pan · Casey Dugan · Zahra Ashktorab · Michelle Brachman -
2020 Expo Talk Panel: Human-Centered AI @ IBM Research –Automation versus Collaboration in the Age of AI »
Werner Geyer · Casey Dugan -
2019 Demonstration: Passcode: A cooperative word guessing game between a human and AI agent »
Katy Gero · Maria Ruis · Zahra Ashktorab · J Johnson · Sadhana Kumaravel · Wei Zhang · Qian Pan · Murray Campbell · Casey Dugan · David Millen · Sarah Miller · Werner Geyer -
2019 Demonstration: "How can this Paper get in?" - A game to advise researchers when writing for a top AI conference »
Aabhas Sharma · Narendra Nath Joshi · Michael Muller · Casey Dugan -
2018 Demonstration: BigBlueBot: A Demonstration of How to Detect Egregious Conversations with Chatbots »
Casey Dugan · Justin D Weisz · Aabhas Sharma · Ingrid Lange · J Johnson · Mohit Jain · Werner Geyer -
2018 Demonstration: Game for Detecting Backdoor Attacks on Deep Neural Networks using Activation Clustering »
Casey Dugan · Werner Geyer · Narendra Nath Joshi · Ingrid Lange · Dustin Ramsey Torres · Bryant Chen · Nathalie Baracaldo · Heiko Ludwig