Timezone: »

 
Spotlight
RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning
Yingbin Bai · Erkun Yang · Zhaoqing Wang · Yuxuan Du · Bo Han · Cheng Deng · Dadong Wang · Tongliang Liu

Wed Dec 07 09:00 AM -- 11:00 AM (PST) @
in Lightning Talks 3B-3 »

Most recent self-supervised learning methods learn visual representation by contrasting different augmented views of images. Compared with supervised learning, more aggressive augmentations have been introduced to further improve the diversity of training pairs. However, aggressive augmentations may distort images' structures leading to a severe semantic shift problem that augmented views of the same image may not share the same semantics, thus degrading the transfer performance. To address this problem, we propose a new SSL paradigm, which counteracts the impact of semantic shift by balancing the role of weak and aggressively augmented pairs. Specifically, semantically inconsistent pairs are of minority, and we treat them as noisy pairs. Note that deep neural networks (DNNs) have a crucial memorization effect that DNNs tend to first memorize clean (majority) examples before overfitting to noisy (minority) examples. Therefore, we set a relatively large weight for aggressively augmented data pairs at the early learning stage. With the training going on, the model begins to overfit noisy pairs. Accordingly, we gradually reduce the weights of aggressively augmented pairs. In doing so, our method can better embrace aggressive augmentations and neutralize the semantic shift problem. Experiments show that our model achieves 73.1% top-1 accuracy on ImageNet-1K with ResNet-50 for 200 epochs, which is a 2.5% improvement over BYOL. Moreover, experiments also demonstrate that the learned representations can transfer well for various downstream tasks. Code is released at: https://github.com/tmllab/RSA.

Author Information

Yingbin Bai (The University of Sydney)
Erkun Yang (Xidian University)
Zhaoqing Wang (The University of Sydney, University of Sydney)
Yuxuan Du (JD explore Academy)
Bo Han (HKBU / RIKEN)
Cheng Deng (Xidian University)
Dadong Wang (CSIRO)
Tongliang Liu (The University of Sydney)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors