Timezone: »

 
Spotlight
Kernel Based Progressive Distillation for Adder Neural Networks
Yixing Xu · Chang Xu · Xinghao Chen · Wei Zhang · Chunjing XU · Yunhe Wang

Thu Dec 10 07:10 PM -- 07:20 PM (PST) @ Orals & Spotlights: Deep Learning
in Orals & Spotlights Track 34: Deep Learning »
Adder Neural Networks (ANNs) which only contain additions bring us a new way of developing deep neural networks with low energy consumption. Unfortunately, there is an accuracy drop when replacing all convolution filters by adder filters. The main reason here is the optimization difficulty of ANNs using $\ell_1$-norm, in which the estimation of gradient in back propagation is inaccurate. In this paper, we present a novel method for further improving the performance of ANNs without increasing the trainable parameters via a progressive kernel based knowledge distillation (PKKD) method. A convolutional neural network (CNN) with the same architecture is simultaneously initialized and trained as a teacher network, features and weights of ANN and CNN will be transformed to a new space to eliminate the accuracy drop. The similarity is conducted in a higher-dimensional space to disentangle the difference of their distributions using a kernel based method. Finally, the desired ANN is learned based on the information from both the ground-truth and teacher, progressively. The effectiveness of the proposed method for learning ANN with higher performance is then well-verified on several benchmarks. For instance, the ANN-50 trained using the proposed PKKD method obtains a 76.8\% top-1 accuracy on ImageNet dataset, which is 0.6\% higher than that of the ResNet-50.
Keywords: Multi-Agent RL · Game Playing · Applications · Reinforcement Learning and Planning

Author Information

Yixing Xu (Huawei Noah's Ark Lab)
Chang Xu (University of Sydney)
Xinghao Chen (Noah's Ark Lab, Huawei Technologies)
Wei Zhang (Noah's Ark Lab, Huawei Inc.)
Chunjing XU (Huawei Technologies)
Yunhe Wang (Huawei Noah's Ark Lab)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors