Timezone: »

 
Poster
Knowledge Diffusion for Distillation
Tao Huang · Yuan Zhang · Mingkai Zheng · Shan You · Fei Wang · Chen Qian · Chang Xu

Tue Dec 12 08:45 AM -- 10:45 AM (PST) @ Great Hall & Hall B1+B2 #207
in Poster Session 1 »
Event URL: https://github.com/hunto/DiffKD »

The representation gap between teacher and student is an emerging topic in knowledge distillation (KD). To reduce the gap and improve the performance, current methods often resort to complicated training schemes, loss functions, and feature alignments, which are task-specific and feature-specific. In this paper, we state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature, and propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models. Our approach is based on the observation that student features typically contain more noises than teacher features due to the smaller capacity of student model. To address this, we propose to denoise student features using a diffusion model trained by teacher features. This allows us to perform better distillation between the refined clean feature and teacher feature. Additionally, we introduce a light-weight diffusion model with a linear autoencoder to reduce the computation cost and an adaptive noise matching module to improve the denoising performance. Extensive experiments demonstrate that DiffKD is effective across various types of features and achieves state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at https://github.com/hunto/DiffKD.

Author Information

Tao Huang (The University of Sydney)

I am a Ph.D. student at The University of Sydney, advised by Dr. Chang Xu. Before that, I was a Researcher at SenseTime. I am actively doing research in the field of efficient deep learning, including knowledge distillation, neural architecture search, structural pruning, e.t.c.

Yuan Zhang (Peking University)
Mingkai Zheng (University of Sydney)
Shan You (SenseTime Research)
Fei Wang (Sensetime)
Chen Qian (SenseTime)
Chang Xu (University of Sydney)

More from the Same Authors