Timezone: »
Multiple lines of evidence suggest that predictive models may benefit from algorithmic triage. Under algorithmic triage, a predictive model does not predict all instances but instead defers some of them to human experts. However, the interplay between the prediction accuracy of the model and the human experts under algorithmic triage is not well understood. In this work, we start by formally characterizing under which circumstances a predictive model may benefit from algorithmic triage. In doing so, we also demonstrate that models trained for full automation may be suboptimal under triage. Then, given any model and desired level of triage, we show that the optimal triage policy is a deterministic threshold rule in which triage decisions are derived deterministically by thresholding the difference between the model and human errors on a per-instance level. Building upon these results, we introduce a practical gradient-based algorithm that is guaranteed to find a sequence of predictive models and triage policies of increasing performance. Experiments on a wide variety of supervised learning tasks using synthetic and real data from two important applications---content moderation and scientific discovery---illustrate our theoretical results and show that the models and triage policies provided by our gradient-based algorithm outperform those provided by several competitive baselines.
Author Information
Nastaran Okati (Max Planck Institute for Software Systems)
Abir De (IIT Bombay)
Manuel Rodriguez (Max Planck Institute for Software Systems)
More from the Same Authors
-
2021 : Reinforcement Learning Under Algorithmic Triage »
Eleni Straitouri · Adish Singla · Vahid Balazadeh Meresht · Manuel Rodriguez -
2022 Poster: Counterfactual Temporal Point Processes »
Kimia Noorbakhsh · Manuel Rodriguez -
2023 Poster: Locality Sensitive Hashing in Fourier Frequency Domain For Soft Set Containment Search »
Indradyumna Roy · Rishi Agarwal · Soumen Chakrabarti · Anirban Dasgupta · Abir De -
2023 Poster: Learning to Select a Subset of Training Examples to Generalize Efficient Model Training »
Eeshaan Jain · Tushar Nandy · Gaurav Aggarwal · Ashish Tendulkar · Rishabh Iyer · Abir De -
2023 Poster: Finding Counterfactually Optimal Action Sequences in Continuous State Spaces »
Stratis Tsirtsis · Manuel Rodriguez -
2023 Poster: Human-Aligned Calibration for AI-Assisted Decision Making »
Nina Corvelo Benz · Manuel Rodriguez -
2022 Spotlight: Neural Estimation of Submodular Functions with Applications to Differentiable Subset Selection »
Abir De · Soumen Chakrabarti -
2022 Spotlight: Lightning Talks 1A-3 »
Kimia Noorbakhsh · Ronan Perry · Qi Lyu · Jiawei Jiang · Christian Toth · Olivier Jeunen · Xin Liu · Yuan Cheng · Lei Li · Manuel Rodriguez · Julius von Kügelgen · Lars Lorch · Nicolas Donati · Lukas Burkhalter · Xiao Fu · Zhongdao Wang · Songtao Feng · Ciarán Gilligan-Lee · Rishabh Mehrotra · Fangcheng Fu · Jing Yang · Bernhard Schölkopf · Ya-Li Li · Christian Knoll · Maks Ovsjanikov · Andreas Krause · Shengjin Wang · Hong Zhang · Mounia Lalmas · Bolin Ding · Bo Du · Yingbin Liang · Franz Pernkopf · Robert Peharz · Anwar Hithnawi · Julius von Kügelgen · Bo Li · Ce Zhang -
2022 Spotlight: Maximum Common Subgraph Guided Graph Retrieval: Late and Early Interaction Networks »
Indradyumna Roy · Soumen Chakrabarti · Abir De -
2022 Spotlight: Counterfactual Temporal Point Processes »
Kimia Noorbakhsh · Manuel Rodriguez -
2022 Spotlight: Learning Recourse on Instance Environment to Enhance Prediction Accuracy »
Lokesh N · Guntakanti Sai Koushik · Abir De · Sunita Sarawagi -
2022 Poster: Neural Estimation of Submodular Functions with Applications to Differentiable Subset Selection »
Abir De · Soumen Chakrabarti -
2022 Poster: Learning Recourse on Instance Environment to Enhance Prediction Accuracy »
Lokesh N · Guntakanti Sai Koushik · Abir De · Sunita Sarawagi -
2022 Poster: Maximum Common Subgraph Guided Graph Retrieval: Late and Early Interaction Networks »
Indradyumna Roy · Soumen Chakrabarti · Abir De -
2021 Workshop: Human Centered AI »
Michael Muller · Plamen P Angelov · Shion Guha · Marina Kogan · Gina Neff · Nuria Oliver · Manuel Rodriguez · Adrian Weller -
2021 Poster: Learning to Select Exogenous Events for Marked Temporal Point Process »
Ping Zhang · Rishabh Iyer · Ashish Tendulkar · Gaurav Aggarwal · Abir De -
2021 Poster: Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time »
Anshul Nasery · Soumyadeep Thakur · Vihari Piratla · Abir De · Sunita Sarawagi -
2021 Poster: Counterfactual Explanations in Sequential Decision Making Under Uncertainty »
Stratis Tsirtsis · Abir De · Manuel Rodriguez -
2019 : Poster Session »
Ayse Cakmak · Yunkai Zhang · Srijith Prabhakarannair Kusumam · Mohamed Osama Ahmed · Xintao Wu · Jayesh Choudhari · David I Inouye · Thomas Taylor · Michel Besserve · Ali Caner Turkmen · Kazi Islam · Antonio Artés · Amrith Setlur · Zhanghua Fu · Zhen Han · Abir De · Nan Du · Pablo Sanchez-Martin -
2019 Workshop: Learning with Temporal Point Processes »
Manuel Rodriguez · Le Song · Isabel Valera · Yan Liu · Abir De · Hongyuan Zha -
2019 Workshop: Workshop on Human-Centric Machine Learning »
Plamen P Angelov · Nuria Oliver · Adrian Weller · Manuel Rodriguez · Isabel Valera · Silvia Chiappa · Hoda Heidari · Niki Kilbertus -
2018 : Manuel Gomez Rodriguez - Enhancing the Accuracy and Fairness of Human Decision Making »
Manuel Rodriguez -
2018 Poster: Deep Reinforcement Learning of Marked Temporal Point Processes »
Utkarsh Upadhyay · Abir De · Manuel Gomez Rodriguez -
2017 Poster: From Parity to Preference-based Notions of Fairness in Classification »
Muhammad Bilal Zafar · Isabel Valera · Manuel Rodriguez · Krishna Gummadi · Adrian Weller -
2016 Poster: Learning and Forecasting Opinion Dynamics in Social Networks »
Abir De · Isabel Valera · Niloy Ganguly · Sourangshu Bhattacharya · Manuel Gomez Rodriguez -
2015 Poster: COEVOLVE: A Joint Point Process Model for Information Diffusion and Network Co-evolution »
Mehrdad Farajtabar · Yichen Wang · Manuel Rodriguez · Shuang Li · Hongyuan Zha · Le Song -
2015 Oral: COEVOLVE: A Joint Point Process Model for Information Diffusion and Network Co-evolution »
Mehrdad Farajtabar · Yichen Wang · Manuel Rodriguez · Shuang Li · Hongyuan Zha · Le Song