Timezone: »
The memorization effect of deep neural network (DNN) plays a pivotal role in many state-of-the-art label-noise learning methods. To exploit this property, the early stopping trick, which stops the optimization at the early stage of training, is usually adopted. Current methods generally decide the early stopping point by considering a DNN as a whole. However, a DNN can be considered as a composition of a series of layers, and we find that the latter layers in a DNN are much more sensitive to label noise, while their former counterparts are quite robust. Therefore, selecting a stopping point for the whole network may make different DNN layers antagonistically affect each other, thus degrading the final performance. In this paper, we propose to separate a DNN into different parts and progressively train them to address this problem. Instead of the early stopping which trains a whole DNN all at once, we initially train former DNN layers by optimizing the DNN with a relatively large number of epochs. During training, we progressively train the latter DNN layers by using a smaller number of epochs with the preceding layers fixed to counteract the impact of noisy labels. We term the proposed method as progressive early stopping (PES). Despite its simplicity, compared with the traditional early stopping, PES can help to obtain more promising and stable results. Furthermore, by combining PES with existing approaches on noisy label training, we achieve state-of-the-art performance on image classification benchmarks. The code is made public at https://github.com/tmllab/PES.
Author Information
Yingbin Bai (The University of Sydney)
Erkun Yang (Xidian University)
Bo Han (HKBU / RIKEN)
Yanhua Yang (Xidian University)
Jiatong Li (University of Technology Sydney)
Yinian Mao (Meituan)
Gang Niu (RIKEN)

Gang Niu is currently an indefinite-term senior research scientist at RIKEN Center for Advanced Intelligence Project.
Tongliang Liu (The University of Sydney)
More from the Same Authors
-
2021 Spotlight: TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptation »
Haoang Chi · Feng Liu · Wenjing Yang · Long Lan · Tongliang Liu · Bo Han · William Cheung · James Kwok -
2021 : On the Role of Pre-training for Meta Few-Shot Learning »
Chia-You Chen · Hsuan-Tien Lin · Masashi Sugiyama · Gang Niu -
2022 Poster: Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses »
Yuzhou Cao · Tianchi Cai · Lei Feng · Lihong Gu · Jinjie GU · Bo An · Gang Niu · Masashi Sugiyama -
2022 Poster: RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning »
Yingbin Bai · Erkun Yang · Zhaoqing Wang · Yuxuan Du · Bo Han · Cheng Deng · Dadong Wang · Tongliang Liu -
2022 Poster: Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs »
Yongqiang Chen · Yonggang Zhang · Yatao Bian · Han Yang · MA Kaili · Binghui Xie · Tongliang Liu · Bo Han · James Cheng -
2022 Poster: Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attacks »
Jianan Zhou · Jianing Zhu · Jingfeng ZHANG · Tongliang Liu · Gang Niu · Bo Han · Masashi Sugiyama -
2022 Poster: Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning »
Shikun Li · Xiaobo Xia · Hansong Zhang · Yibing Zhan · Shiming Ge · Tongliang Liu -
2022 Poster: Towards Lightweight Black-Box Attack Against Deep Neural Networks »
Chenghao Sun · Yonggang Zhang · Wan Chaoqun · Qizhou Wang · Ya Li · Tongliang Liu · Bo Han · Xinmei Tian -
2022 : Pre-training Robust Feature Extractor Against Clean-label Data Poisoning Attacks »
Ting Zhou · Hanshu Yan · Lei LIU · Jingfeng Zhang · Bo Han -
2022 Spotlight: Lightning Talks 6A-4 »
Xiu-Shen Wei · Konstantina Dritsa · Guillaume Huguet · ABHRA CHAUDHURI · Zhenbin Wang · Kevin Qinghong Lin · Yutong Chen · Jianan Zhou · Yongsen Mao · Junwei Liang · Jinpeng Wang · Mao Ye · Yiming Zhang · Aikaterini Thoma · H.-Y. Xu · Daniel Sumner Magruder · Enwei Zhang · Jianing Zhu · Ronglai Zuo · Massimiliano Mancini · Hanxiao Jiang · Jun Zhang · Fangyun Wei · Faen Zhang · Ioannis Pavlopoulos · Zeynep Akata · Xiatian Zhu · Jingfeng ZHANG · Alexander Tong · Mattia Soldan · Chunhua Shen · Yuxin Peng · Liuhan Peng · Michael Wray · Tongliang Liu · Anjan Dutta · Yu Wu · Oluwadamilola Fasina · Panos Louridas · Angel Chang · Manik Kuchroo · Manolis Savva · Shujie LIU · Wei Zhou · Rui Yan · Gang Niu · Liang Tian · Bo Han · Eric Z. XU · Guy Wolf · Yingying Zhu · Brian Mak · Difei Gao · Masashi Sugiyama · Smita Krishnaswamy · Rong-Cheng Tu · Wenzhe Zhao · Weijie Kong · Chengfei Cai · WANG HongFa · Dima Damen · Bernard Ghanem · Wei Liu · Mike Zheng Shou -
2022 Spotlight: Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attacks »
Jianan Zhou · Jianing Zhu · Jingfeng ZHANG · Tongliang Liu · Gang Niu · Bo Han · Masashi Sugiyama -
2022 Spotlight: Lightning Talks 5B-3 »
Yanze Wu · Jie Xiao · Nianzu Yang · Jieyi Bi · Jian Yao · Yiting Chen · Qizhou Wang · Yangru Huang · Yongqiang Chen · Peixi Peng · Yuxin Hong · Xintao Wang · Feng Liu · Yining Ma · Qibing Ren · Xueyang Fu · Yonggang Zhang · Kaipeng Zeng · Jiahai Wang · GEN LI · Yonggang Zhang · Qitian Wu · Yifan Zhao · Chiyu Wang · Junchi Yan · Feng Wu · Yatao Bian · Xiaosong Jia · Ying Shan · Zhiguang Cao · Zheng-Jun Zha · Guangyao Chen · Tianjun Xiao · Han Yang · Jing Zhang · Jinbiao Chen · MA Kaili · Yonghong Tian · Junchi Yan · Chen Gong · Tong He · Binghui Xie · Yuan Sun · Francesco Locatello · Tongliang Liu · Yeow Meng Chee · David P Wipf · Tongliang Liu · Bo Han · Bo Han · Yanwei Fu · James Cheng · Zheng Zhang -
2022 Spotlight: Watermarking for Out-of-distribution Detection »
Qizhou Wang · Feng Liu · Yonggang Zhang · Jing Zhang · Chen Gong · Tongliang Liu · Bo Han -
2022 Spotlight: Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs »
Yongqiang Chen · Yonggang Zhang · Yatao Bian · Han Yang · MA Kaili · Binghui Xie · Tongliang Liu · Bo Han · James Cheng -
2022 Spotlight: Lightning Talks 4A-2 »
Barakeel Fanseu Kamhoua · Hualin Zhang · Taiki Miyagawa · Tomoya Murata · Xin Lyu · Yan Dai · Elena Grigorescu · Zhipeng Tu · Lijun Zhang · Taiji Suzuki · Wei Jiang · Haipeng Luo · Lin Zhang · Xi Wang · Young-San Lin · Huan Xiong · Liyu Chen · Bin Gu · Jinfeng Yi · Yongqiang Chen · Sandeep Silwal · Yiguang Hong · Maoyuan Song · Lei Wang · Tianbao Yang · Han Yang · MA Kaili · Samson Zhou · Deming Yuan · Bo Han · Guodong Shi · Bo Li · James Cheng -
2022 Spotlight: Exact Shape Correspondence via 2D graph convolution »
Barakeel Fanseu Kamhoua · Lin Zhang · Yongqiang Chen · Han Yang · MA Kaili · Bo Han · Bo Li · James Cheng -
2022 Spotlight: RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning »
Yingbin Bai · Erkun Yang · Zhaoqing Wang · Yuxuan Du · Bo Han · Cheng Deng · Dadong Wang · Tongliang Liu -
2022 Spotlight: Lightning Talks 2B-4 »
Feiyi Xiao · Amrutha Saseendran · Kwangho Kim · Keyu Yan · Changjian Shui · Guangxi Li · Shikun Li · Edward Kennedy · Man Zhou · Gezheng Xu · Ruilin Ye · Xiaobo Xia · Junjie Tang · Kathrin Skubch · Stefan Falkner · Hansong Zhang · Jose Zubizarreta · Huaying Fang · Xuanqiang Zhao · Jie Huang · Qi CHEN · Yibing Zhan · Jiaqi Li · Xin Wang · Ruibin Xi · Feng Zhao · Margret Keuper · Charles Ling · Shiming Ge · Chengjun Xie · Tongliang Liu · Tal Arbel · Chongyi Li · Danfeng Hong · Boyu Wang · Christian Gagné -
2022 Spotlight: Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning »
Shikun Li · Xiaobo Xia · Hansong Zhang · Yibing Zhan · Shiming Ge · Tongliang Liu -
2022 Poster: MissDAG: Causal Discovery in the Presence of Missing Data with Continuous Additive Noise Models »
Erdun Gao · Ignavier Ng · Mingming Gong · Li Shen · Wei Huang · Tongliang Liu · Kun Zhang · Howard Bondell -
2022 Poster: Watermarking for Out-of-distribution Detection »
Qizhou Wang · Feng Liu · Yonggang Zhang · Jing Zhang · Chen Gong · Tongliang Liu · Bo Han -
2022 Poster: Exact Shape Correspondence via 2D graph convolution »
Barakeel Fanseu Kamhoua · Lin Zhang · Yongqiang Chen · Han Yang · MA Kaili · Bo Han · Bo Li · James Cheng -
2022 Poster: Counterfactual Fairness with Partially Known Causal Graph »
Aoqi Zuo · Susan Wei · Tongliang Liu · Bo Han · Kun Zhang · Mingming Gong -
2022 Poster: Fast and Robust Rank Aggregation against Model Misspecification »
YUANGANG PAN · Ivor W. Tsang · Weijie Chen · Gang Niu · Masashi Sugiyama -
2022 Poster: MetricFormer: A Unified Perspective of Correlation Exploring in Similarity Learning »
Jiexi Yan · Erkun Yang · Cheng Deng · Heng Huang -
2022 Poster: Out-of-Distribution Detection with An Adaptive Likelihood Ratio on Informative Hierarchical VAE »
Yewen Li · Chaojie Wang · Xiaobo Xia · Tongliang Liu · xin miao · Bo An -
2022 Poster: Class-Dependent Label-Noise Learning with Cycle-Consistency Regularization »
De Cheng · Yixiong Ning · Nannan Wang · Xinbo Gao · Heng Yang · Yuxuan Du · Bo Han · Tongliang Liu -
2022 Poster: Synergy-of-Experts: Collaborate to Improve Adversarial Robustness »
Sen Cui · Jingfeng ZHANG · Jian Liang · Bo Han · Masashi Sugiyama · Changshui Zhang -
2022 Poster: Pluralistic Image Completion with Gaussian Mixture Models »
Xiaobo Xia · Wenhao Yang · Jie Ren · Yewen Li · Yibing Zhan · Bo Han · Tongliang Liu -
2022 Poster: Learning Contrastive Embedding in Low-Dimensional Space »
Shuo Chen · Chen Gong · Jun Li · Jian Yang · Gang Niu · Masashi Sugiyama -
2022 Poster: Is Out-of-Distribution Detection Learnable? »
Zhen Fang · Yixuan Li · Jie Lu · Jiahua Dong · Bo Han · Feng Liu -
2021 Poster: Universal Semi-Supervised Learning »
Zhuo Huang · Chao Xue · Bo Han · Jian Yang · Chen Gong -
2021 Poster: Probabilistic Margins for Instance Reweighting in Adversarial Training »
qizhou wang · Feng Liu · Bo Han · Tongliang Liu · Chen Gong · Gang Niu · Mingyuan Zhou · Masashi Sugiyama -
2021 Poster: Instance-dependent Label-noise Learning under a Structural Causal Model »
Yu Yao · Tongliang Liu · Mingming Gong · Bo Han · Gang Niu · Kun Zhang -
2021 Poster: TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptation »
Haoang Chi · Feng Liu · Wenjing Yang · Long Lan · Tongliang Liu · Bo Han · William Cheung · James Kwok -
2021 Poster: Confident Anchor-Induced Multi-Source Free Domain Adaptation »
Jiahua Dong · Zhen Fang · Anjin Liu · Gan Sun · Tongliang Liu -
2020 Poster: Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning »
Yu Yao · Tongliang Liu · Bo Han · Mingming Gong · Jiankang Deng · Gang Niu · Masashi Sugiyama -
2020 Poster: Part-dependent Label Noise: Towards Instance-dependent Label Noise »
Xiaobo Xia · Tongliang Liu · Bo Han · Nannan Wang · Mingming Gong · Haifeng Liu · Gang Niu · Dacheng Tao · Masashi Sugiyama -
2020 Spotlight: Part-dependent Label Noise: Towards Instance-dependent Label Noise »
Xiaobo Xia · Tongliang Liu · Bo Han · Nannan Wang · Mingming Gong · Haifeng Liu · Gang Niu · Dacheng Tao · Masashi Sugiyama -
2020 Poster: Rethinking Importance Weighting for Deep Learning under Distribution Shift »
Tongtong Fang · Nan Lu · Gang Niu · Masashi Sugiyama -
2020 Spotlight: Rethinking Importance Weighting for Deep Learning under Distribution Shift »
Tongtong Fang · Nan Lu · Gang Niu · Masashi Sugiyama -
2020 Poster: Provably Consistent Partial-Label Learning »
Lei Feng · Jiaqi Lv · Bo Han · Miao Xu · Gang Niu · Xin Geng · Bo An · Masashi Sugiyama -
2020 Poster: Domain Generalization via Entropy Regularization »
Shanshan Zhao · Mingming Gong · Tongliang Liu · Huan Fu · Dacheng Tao -
2019 : Poster Presentations »
Rahul Mehta · Andrew Lampinen · Binghong Chen · Sergio Pascual-Diaz · Jordi Grau-Moya · Aldo Faisal · Jonathan Tompson · Yiren Lu · Khimya Khetarpal · Martin Klissarov · Pierre-Luc Bacon · Doina Precup · Thanard Kurutach · Aviv Tamar · Pieter Abbeel · Jinke He · Maximilian Igl · Shimon Whiteson · Wendelin Boehmer · Raphaël Marinier · Olivier Pietquin · Karol Hausman · Sergey Levine · Chelsea Finn · Tianhe Yu · Lisa Lee · Benjamin Eysenbach · Emilio Parisotto · Eric Xing · Ruslan Salakhutdinov · Hongyu Ren · Anima Anandkumar · Deepak Pathak · Christopher Lu · Trevor Darrell · Alexei Efros · Phillip Isola · Feng Liu · Bo Han · Gang Niu · Masashi Sugiyama · Saurabh Kumar · Janith Petangoda · Johan Ferret · James McClelland · Kara Liu · Animesh Garg · Robert Lange -
2019 Poster: Uncoupled Regression from Pairwise Comparison Data »
Ritsugen Jo · Junya Honda · Gang Niu · Masashi Sugiyama -
2019 Poster: Are Anchor Points Really Indispensable in Label-Noise Learning? »
Xiaobo Xia · Tongliang Liu · Nannan Wang · Bo Han · Chen Gong · Gang Niu · Masashi Sugiyama -
2019 Poster: Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence »
Fengxiang He · Tongliang Liu · Dacheng Tao -
2018 Poster: Binary Classification from Positive-Confidence Data »
Takashi Ishida · Gang Niu · Masashi Sugiyama -
2018 Spotlight: Binary Classification from Positive-Confidence Data »
Takashi Ishida · Gang Niu · Masashi Sugiyama -
2018 Poster: Masking: A New Perspective of Noisy Supervision »
Bo Han · Jiangchao Yao · Gang Niu · Mingyuan Zhou · Ivor Tsang · Ya Zhang · Masashi Sugiyama -
2018 Poster: Co-teaching: Robust training of deep neural networks with extremely noisy labels »
Bo Han · Quanming Yao · Xingrui Yu · Gang Niu · Miao Xu · Weihua Hu · Ivor Tsang · Masashi Sugiyama -
2017 Poster: Positive-Unlabeled Learning with Non-Negative Risk Estimator »
Ryuichi Kiryo · Gang Niu · Marthinus C du Plessis · Masashi Sugiyama -
2017 Poster: Learning from Complementary Labels »
Takashi Ishida · Gang Niu · Weihua Hu · Masashi Sugiyama -
2017 Oral: Positive-Unlabeled Learning with Non-Negative Risk Estimator »
Ryuichi Kiryo · Gang Niu · Marthinus C du Plessis · Masashi Sugiyama -
2016 Poster: Theoretical Comparisons of Positive-Unlabeled Learning against Positive-Negative Learning »
Gang Niu · Marthinus Christoffel du Plessis · Tomoya Sakai · Yao Ma · Masashi Sugiyama -
2014 Poster: Analysis of Learning from Positive and Unlabeled Data »
Marthinus C du Plessis · Gang Niu · Masashi Sugiyama -
2011 Poster: Analysis and Improvement of Policy Gradient Estimation »
Tingting Zhao · Hirotaka Hachiya · Gang Niu · Masashi Sugiyama