Timezone: »

 
Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge
ZIYUN LI · Xinshao Wang · Christoph Meinel · Neil Robertson · David Clifton · Haojin Yang

@
in Trustworthy and Socially Responsible Machine Learning »
Event URL: https://openreview.net/forum?id=HHIFHJtJeGA »

Mutual knowledge distillation (MKD) improves a model by distilling knowledge from another model. However, \textit{not all knowledge is certain and correct}, especially under adverse conditions.For example, label noise usually leads to less reliable models due to undesired memorization. Wrong knowledge harms the learning rather than helps it. This problem can be handled by two aspects: (i) knowledge source, improving the reliability of each model (knowledge producer) improving the knowledge source's reliability; (ii) selecting reliable knowledge for distillation. Making a model more reliable is widely studied while selective MKD receives little attention. Therefore, we focus on studying selective MKD and highlight its importance in this work. Concretely, a generic MKD framework, \underline{C}onfident knowledge selection followed by \underline{M}utual \underline{D}istillation (CMD), is designed. The key component of CMD is a generic knowledge selection formulation, making the selection threshold either static (CMD-S) or progressive (CMD-P). Additionally, CMD covers two special cases: zero knowledge and all knowledge, leading to a unified MKD framework. Extensive experiments are present to demonstrate the effectiveness of CMD and thoroughly justify the design of CMD.

[ Poster]  [ OpenReview]  [ Topia] 

Author Information

ZIYUN LI (Hasso Plattner Institut)
Xinshao Wang (University of Oxford)
Christoph Meinel (Hasso Plattner Institute)
Neil Robertson (Queen's University Belfast)
David Clifton (University of Oxford)

Professor of Clinical Machine Learning Department of Engineering Science University of Oxford

Haojin Yang (Hasso-Plattner-Institut für Digital Engineering gGmbH)

More from the Same Authors