`

Timezone: »

 
User-in-the-Loop Named Entity Recognition via Counterfactual Learning
Tong Yu · Junda Wu · Ruiyi Zhang · Handong Zhao · Shuai Li

Named Entity Recognition (NER) is an important task, to enable a wide range of NLP applications. The state-of-the-art NER models are based on deep learning and require enough labeled data. In practice, the labeled data for NER is usually limited, as providing accurate labels to the sentences is very time consuming. With an NER model trained on limited labeled data, it is desirable to develop an efficient mechanism to collect data labels and improve the model over time. To achieve this, existing works develop active learning approaches. However, these approaches are usually developed for annotators and assume the annotators will provide the exactly correct labels to each sentence selected to label. In this paper, we propose a simple yet effective user-in-the-loop feedback mechanism to enable end users, instead of annotators, to easily provide labels to the system. We identify counterfactual bias of the data collected by this feedback mechanism. To alleviate the bias and achieve more sample-efficient learning, we further develop a counterfactual NER learning framework. We develop an imputation model to estimate the loss in those non-displayed entity classes. By considering both losses on displayed and non-displayed entity classes, we can efficiently alleviate such display bias in the NER model. With extensive experiments, we validate the effectiveness of our feedback mechanism and learning framework.

Author Information

Tong Yu (Adobe Research)
Junda Wu (New York University)
Ruiyi Zhang (Adobe Research)
Handong Zhao (Adobe Research)
Shuai Li (Shanghai Jiao Tong University)

More from the Same Authors

  • 2021 : Sim-to-Real Interactive Recommendation via Off-Dynamics Reinforcement Learning »
    Junda Wu · Zhihui Xie · Tong Yu · Qizhi Li · Shuai Li
  • 2021 Poster: Slow Learning and Fast Inference: Efficient Graph Similarity Computation via Knowledge Distillation »
    Can Qin · Handong Zhao · Lichen Wang · Huan Wang · Yulun Zhang · Yun Fu
  • 2021 Poster: Understanding Bandits with Graph Feedback »
    Houshuang Chen · zengfeng Huang · Shuai Li · Chihao Zhang
  • 2021 Poster: The Hardness Analysis of Thompson Sampling for Combinatorial Semi-bandits with Greedy Oracle »
    Fang Kong · Yueran Yang · Wei Chen · Shuai Li
  • 2021 Poster: UniDoc: Unified Pretraining Framework for Document Understanding »
    Jiuxiang Gu · Jason Kuen · Vlad I Morariu · Handong Zhao · Rajiv Jain · Nikolaos Barmpalios · Ani Nenkova · Tong Sun
  • 2020 Poster: Online Influence Maximization under Linear Threshold Model »
    Shuai Li · Fang Kong · Kejie Tang · Qizhi Li · Wei Chen
  • 2020 Poster: Self-Supervised Relationship Probing »
    Jiuxiang Gu · Jason Kuen · Shafiq Joty · Jianfei Cai · Vlad I. Morariu · Handong Zhao · Tong Sun
  • 2019 : Poster and Coffee Break 2 »
    Karol Hausman · Kefan Dong · Ken Goldberg · Lihong Li · Lin Yang · Lingxiao Wang · Lior Shani · Liwei Wang · Loren Amdahl-Culleton · Lucas Cassano · Marc Dymetman · Marc Bellemare · Marcin Tomczak · Margarita Castro · Marius Kloft · Marius-Constantin Dinu · Markus Holzleitner · Martha White · Mengdi Wang · Michael Jordan · Mihailo Jovanovic · Ming Yu · Minshuo Chen · Moonkyung Ryu · Muhammad Zaheer · Naman Agarwal · Nan Jiang · Niao He · Nikolaus Yasui · Nikos Karampatziakis · Nino Vieillard · Ofir Nachum · Olivier Pietquin · Ozan Sener · Pan Xu · Parameswaran Kamalaruban · Paul Mineiro · Paul Rolland · Philip Amortila · Pierre-Luc Bacon · Prakash Panangaden · Qi Cai · Qiang Liu · Quanquan Gu · Raihan Seraj · Richard Sutton · Rick Valenzano · Robert Dadashi · Rodrigo Toro Icarte · Roshan Shariff · Roy Fox · Ruosong Wang · Saeed Ghadimi · Samuel Sokota · Sean Sinclair · Sepp Hochreiter · Sergey Levine · Sergio Valcarcel Macua · Sham Kakade · Shangtong Zhang · Sheila McIlraith · Shie Mannor · Shimon Whiteson · Shuai Li · Shuang Qiu · Wai Lok Li · Siddhartha Banerjee · Sitao Luan · Tamer Basar · Thinh Doan · Tianhe Yu · Tianyi Liu · Tom Zahavy · Toryn Klassen · Tuo Zhao · Vicenç Gómez · Vincent Liu · Volkan Cevher · Wesley Suttle · Xiao-Wen Chang · Xiaohan Wei · Xiaotong Liu · Xingguo Li · Xinyi Chen · Xingyou Song · Yao Liu · YiDing Jiang · Yihao Feng · Yilun Du · Yinlam Chow · Yinyu Ye · Yishay Mansour · · Yonathan Efroni · Yongxin Chen · Yuanhao Wang · Bo Dai · Chen-Yu Wei · Harsh Shrivastava · Hongyang Zhang · Qinqing Zheng · SIDDHARTHA SATPATHI · Xueqing Liu · Andreu Vall
  • 2018 Poster: TopRank: A practical algorithm for online stochastic ranking »
    Tor Lattimore · Branislav Kveton · Shuai Li · Csaba Szepesvari