Timezone: »
Human-in-the-loop data labeling is generally considered a tedious, error-prone and expensive activity. Automation of the labeling task is desirable, but current approaches can conflict with principles of trust and human agency. We are developing a data labeling experience where the human labeler transparently interacts with an AI assistant to reach automation readiness, at which point the remainder of the labeling task can be delegated to a virtual assistant. Our approach combines semi-supervised learning, active learning, and human-machine decision tracking to reduce labeling effort and support reliable automation. The demo takes participant through an online end-to-end AI assisted data labeling experience, starting with manual labeling, then assisted labeling and ultimately transitioning to automated labeling via a system of readiness checkpoints.
Author Information
Michael Desmond (IBM Research AI)
Evelyn Duesterwald (IBM Research)
Kristina Brimijoin (IBM Research)
Michael Muller (IBM Research)
Michael Muller works in the AI Interactions group of IBM Research AI, where his work focuses on the human aspects of data science; ethics and values in applications of AI to human issues; metrics and analytics for enterprise social software applications, with particular application to employee engagement emergent social phenomena in social software. Recognitions include: ACM Distinguished Scientist; SIGCHI Academy; IBM Master Inventor. Steering Committees: EUSSET (European Society for the study of Socially Embedded Technologies); ACM GROUP conference series. Papers co-chair for ECSCW 2019 (European Computer Supported Cooperative Work conference).
Aabhas Sharma (IBM Research)
Narendra Nath Joshi (IBM Research)
Qian Pan (IBM Research AI)
Casey Dugan (IBM Research AI)
Zahra Ashktorab (IBM Research AI)
Michelle Brachman (IBM Research AI)
More from the Same Authors
-
2022 Workshop: NeurIPS 2022 Workshop Virtual Proposal: HCAI@NeurIPS 2022, Human Centered AI »
Michael Muller · Plamen P Angelov · Hal Daumé III · Shion Guha · Q.Vera Liao · Nuria Oliver · David Piorkowski -
2021 Workshop: Human Centered AI »
Michael Muller · Plamen P Angelov · Shion Guha · Marina Kogan · Gina Neff · Nuria Oliver · Manuel Rodriguez · Adrian Weller -
2020 Demonstration: Fast and Automatic Visual Label Conflict Resolution »
Narendra Nath Joshi · Aabhas Sharma · Michelle Brachman · Qian Pan · Michael Muller · Michael Desmond · Kristina Brimijoin · Zahra Ashktorab · Evelyn Duesterwald · Casey Dugan -
2020 Expo Talk Panel: Human-Centered AI @ IBM Research –Automation versus Collaboration in the Age of AI »
Werner Geyer · Casey Dugan -
2019 Demonstration: Passcode: A cooperative word guessing game between a human and AI agent »
Katy Gero · Maria Ruis · Zahra Ashktorab · J Johnson · Sadhana Kumaravel · Wei Zhang · Qian Pan · Murray Campbell · Casey Dugan · David Millen · Sarah Miller · Werner Geyer -
2019 Demonstration: "How can this Paper get in?" - A game to advise researchers when writing for a top AI conference »
Aabhas Sharma · Narendra Nath Joshi · Michael Muller · Casey Dugan -
2018 Demonstration: BigBlueBot: A Demonstration of How to Detect Egregious Conversations with Chatbots »
Casey Dugan · Justin D Weisz · Aabhas Sharma · Ingrid Lange · J Johnson · Mohit Jain · Werner Geyer -
2018 Demonstration: Game for Detecting Backdoor Attacks on Deep Neural Networks using Activation Clustering »
Casey Dugan · Werner Geyer · Narendra Nath Joshi · Ingrid Lange · Dustin Ramsey Torres · Bryant Chen · Nathalie Baracaldo Angel · Heiko Ludwig