What is the role of unlabeled data in an inference problem, when the presumed underlying distribution is adversarially perturbed? To provide a concrete answer to this question, this paper unifies two major learning frameworks: Semi-Supervised Learning (SSL) and Distributionally Robust Learning (DRL). We develop a generalization theory for our framework based on a number of novel complexity measures, such as an adversarial extension of Rademacher complexity and its semi-supervised analogue. Moreover, our analysis is able to quantify the role of unlabeled data in the generalization under a more general condition compared to the existing theoretical works in SSL. Based on our framework, we also present a hybrid of DRL and EM algorithms that has a guaranteed convergence rate. When implemented with deep neural networks, our method shows a comparable performance to those of the state-of-the-art on a number of real-world benchmark datasets.
Amir Najafi (Sharif University of Technology)
Amir Najafi received his B.Sc. and M.Sc. degrees in Electrical Engineering from Sharif University of Technology, Tehran, Iran, in 2012 and 2015, respectively. He is currently a Ph.D. student at Computer Engineering Dept. of Sharif University of Technology. From September 2015 to April 2016, he was with the Broad Institute of MIT and Harvard, MA, USA, as a visiting research scholar. In summer 2018, he was with Preferred Networks Inc., Tokyo, Japan, as an intern. His research interests include machine learning theory, information theory and bioinformatics.
Shin-ichi Maeda (Preferred Networks)
Masanori Koyama (Preferred Networks Inc. )
Takeru Miyato (Preferred Networks, Inc.)
More from the Same Authors
2021 : Contrastive Representation Learning with Trainable Augmentation Channel »
Masanori Koyama · Kentaro Minami · Takeru Miyato · Yarin Gal
2019 Poster: A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation »
Mitsuru Kusumoto · Takuya Inoue · Gentaro Watanabe · Takuya Akiba · Masanori Koyama
2019 Poster: Einconv: Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks »
Kohei Hayashi · Taiki Yamaguchi · Yohei Sugawara · Shin-ichi Maeda