Knowledge distillation (KD) has recently been identified as a method that can unintentionally leak private information regarding the details of a teacher model to an unauthorized student. Recent research in developing undistillable nasty teachers that can protect model confidentiality has gained significant attention. However, the level of protection these nasty models offer has been largely untested. In this paper, we show that transferring knowledge to a shallow sub-section of a student can largely reduce a teacher’s influence. By exploring the depth of the shallow subsection, we then present a distillation technique that enables a skeptical student model to learn even from a nasty teacher. To evaluate the efficacy of our skeptical students, we conducted experiments with several models with KD on both training data-available and data-free scenarios for various datasets. While distilling from nasty teachers, compared to the normal student models, skeptical students consistently provide superior classification performance of up to ∼59.5%. Moreover, similar to normal students, skeptical students maintain high classification accuracy when distilled from a normal teacher, showing their efficacy irrespective of the teacher being nasty or not. We believe the ability of skeptical students to largely diminish the KD-immunity of potentially nasty teachers will motivate the research community to create more robust mechanisms for model confidentiality. We have open-sourced the code at https://github.com/ksouvik52/Skeptical2021
Souvik Kundu (University of Southern California)
Hi! I am a final year Ph.D. student in Electrical & Computer Engineering at **University of Southern California** working under the supervision of Prof. **Massoud Pedram** ([SPORT Lab](http://www.mpedram.com/)) and Prof. **Peter A. Beerel** ([HAL@usc](https://hal.usc.edu/)) . I am an Indian by birth. My current highest academic qualification is M.Tech in Electronics (VLSI) Engineering from **Indian Institute of Technology, Kharagpur**. My technical interests include algorithm-hardware co-design for energy-efficient, robust and privacy-preserving ML for both CMOS and beyond-CMOS technologies using conventional and non-conventional computing pradigm.