Timezone: »
A Simple Framework for Active Learning to Rank
Qingzhong Wang · Haifang Li · Haoyi Xiong · Wen Wang · Jiang Bian · Yu Lu · Shuaiqiang Wang · zhicong cheng · Dawei Yin · Dejing Dou
Learning to rank (LTR) plays a critical role in search engine---there needs to timely label an extremely large number of queries together with relevant webpages to train and update the online LTR models. To reduce the costs and time consumption of queries/webpages labeling, we study the problem of \emph{Active Learning to Rank} (\emph{\bf active LTR}) that selects unlabeled queries for annotation and training in this work. Specifically, we first investigate the criterion--\emph{Ranking Entropy (RE)} characterizing the entropy of relevant webpages under a query produced by a sequence of online LTR models updated by different checkpoints, using a Query-By-Committee (QBC) method. Then, we explore a new criterion namely \emph{Prediction Variances (PV)} that measures the variance of prediction results for all relevant webpages under a query. Our empirical studies find that RE may favor low-frequency queries from the pool for labeling while PV prioritizing high-frequency queries more. Finally, we combine these two complementary criteria as the sample selection strategies for active learning. Extensive experiments with comparisons to baseline algorithms show that the proposed approach could train LTR models achieving higher Discounted Cumulative Gain (\ie, the relative improvement $\Delta$DCG$_4$=1.38\%) with the same budgeted labeling efforts, while the proposed strategies could discover 43\% more valid training pairs for effective training.
Author Information
Qingzhong Wang (Department of Computer Science, City University of Hong Kong)
Haifang Li (Baidu)
Haoyi Xiong
Wen Wang (Baidu)
Jiang Bian (Baidu Inc. )
Yu Lu (Baidu)
Shuaiqiang Wang (Baidu Inc.)
zhicong cheng (BAIDU)
Dawei Yin (jd)
Dejing Dou (Baidu)
More from the Same Authors
-
2022 Poster: Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement »
Yan Li · Xinjiang Lu · Yaqing Wang · Dejing Dou -
2022 Poster: InterpretDL: Explaining Deep Models in PaddlePaddle »
Xuhong Li · Haoyi Xiong · Xingjian Li · Xuanyu Wu · Zeyu Chen · Dejing Dou -
2022 : SMILE: Sample-to-feature MIxup for Efficient Transfer LEarning »
Xingjian Li · Haoyi Xiong · Cheng-Zhong Xu · Dejing Dou -
2022 : A Comparative Survey of Deep Active Learning »
Xueying Zhan · Qingzhong Wang · Kuan-Hao Huang · Haoyi Xiong · Dejing Dou · Antoni Chan -
2023 Poster: Learning to Tokenize for Generative Retrieval »
Weiwei Sun · Lingyong Yan · Zheng Chen · Shuaiqiang Wang · Haichao Zhu · Pengjie Ren · Zhumin Chen · Dawei Yin · Maarten Rijke · Zhaochun Ren -
2023 Poster: Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking »
Juanhui Li · Harry Shomer · Haitao Mao · Shenglai Zeng · Yao Ma · Neil Shah · Jiliang Tang · Dawei Yin -
2022 Spotlight: InterpretDL: Explaining Deep Models in PaddlePaddle »
Xuhong Li · Haoyi Xiong · Xingjian Li · Xuanyu Wu · Zeyu Chen · Dejing Dou -
2022 Spotlight: Lightning Talks 1A-1 »
Siba Smarak Panigrahi · Xuhong Li · Mikhail Usvyatsov · Shaohan Chen · Sohan Patnaik · Haoyi Xiong · Nikolaos V Sahinidis · Rafael Ballester-Ripoll · Chuanhou Gao · Xingjian Li · Konrad Schindler · Xuanyu Wu · Zeyu Chen · Dejing Dou -
2022 Poster: A Large Scale Search Dataset for Unbiased Learning to Rank »
Lixin Zou · Haitao Mao · Xiaokai Chu · Jiliang Tang · Wenwen Ye · Shuaiqiang Wang · Dawei Yin -
2022 Poster: AutoMS: Automatic Model Selection for Novelty Detection with Error Rate Control »
Yifan Zhang · Haiyan Jiang · Haojie Ren · Changliang Zou · Dejing Dou -
2021 : [O6] Explaining Information Flow Inside Vision Transformers Using Markov Chain »
Tingyi Yuan · Xuhong Li · Haoyi Xiong · Dejing Dou -
2020 Poster: Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching »
Di Hu · Rui Qian · Minyue Jiang · Xiao Tan · Shilei Wen · Errui Ding · Weiyao Lin · Dejing Dou