Timezone: »

 
Spotlight
PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
Weihan Cao · Yifan Zhang · Jianfei Gao · Anda Cheng · Ke Cheng · Jian Cheng

Wed Dec 07 09:00 AM -- 11:00 AM (PST) @
in Lightning Talks 3B-4 »
Knowledge distillation(KD) is a widely-used technique to train compact models in object detection. However, there is still a lack of study on how to distill between heterogeneous detectors. In this paper, we empirically find that better FPN features from a heterogeneous teacher detector can help the student although their detection heads and label assignments are different. However, directly aligning the feature maps to distill detectors suffers from two problems. First, the difference in feature magnitude between the teacher and the student could enforce overly strict constraints on the student. Second, the FPN stages and channels with large feature magnitude from the teacher model could dominate the gradient of distillation loss, which will overwhelm the effects of other features in KD and introduce much noise. To address the above issues, we propose to imitate features with Pearson Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features. Our method consistently outperforms the existing detection KD methods and works for both homogeneous and heterogeneous student-teacher pairs. Furthermore, it converges faster. With a powerful MaskRCNN-Swin detector as the teacher, ResNet-50 based RetinaNet and FCOS achieve 41.5% and 43.9% $mAP$ on COCO2017, which are 4.1% and 4.8% higher than the baseline, respectively.
Keywords: Object Detection · knowledge distillation

Author Information

Weihan Cao (Institute of Automation Chinese Academy of Sciences)
Yifan Zhang (Institute of automation, Chinese academy of science, Chinese Academy of Sciences)
Jianfei Gao (Southeast University)
Anda Cheng (Institute of automation, Chinese academy of science, Chinese Academy of Sciences)
Ke Cheng (Institute of automation, Chinese academy of science, Chinese Academy of Sciences)
Jian Cheng (Institute of Automation, Chinese Academy of Sciences)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors