Timezone: »

Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
Kento Nozawa · Issei Sato

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @ Virtual

Instance discriminative self-supervised representation learning has been attracted attention thanks to its unsupervised nature and informative feature representation for downstream tasks. In practice, it commonly uses a larger number of negative samples than the number of supervised classes. However, there is an inconsistency in the existing analysis; theoretically, a large number of negative samples degrade classification performance on a downstream supervised task, while empirically, they improve the performance. We provide a novel framework to analyze this empirical result regarding negative samples using the coupon collector's problem. Our bound can implicitly incorporate the supervised loss of the downstream task in the self-supervised loss by increasing the number of negative samples. We confirm that our proposed analysis holds on real-world benchmark datasets.

Author Information

Kento Nozawa (The University of Tokyo / RIKEN)
Issei Sato (The University of Tokyo/RIKEN)

More from the Same Authors