Timezone: »

 
Spotlight
MetaMask: Revisiting Dimensional Confounder for Self-Supervised Learning
Jiangmeng Li · Wenwen Qiang · Yanan Zhang · Wenyi Mo · Changwen Zheng · Bing Su · Hui Xiong

Tue Dec 06 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 2B-3 »

As a successful approach to self-supervised learning, contrastive learning aims to learn invariant information shared among distortions of the input sample. While contrastive learning has yielded continuous advancements in sampling strategy and architecture design, it still remains two persistent defects: the interference of task-irrelevant information and sample inefficiency, which are related to the recurring existence of trivial constant solutions. From the perspective of dimensional analysis, we find out that the dimensional redundancy and dimensional confounder are the intrinsic issues behind the phenomena, and provide experimental evidence to support our viewpoint. We further propose a simple yet effective approach MetaMask, short for the dimensional Mask learned by Meta-learning, to learn representations against dimensional redundancy and confounder. MetaMask adopts the redundancy-reduction technique to tackle the dimensional redundancy issue and innovatively introduces a dimensional mask to reduce the gradient effects of specific dimensions containing the confounder, which is trained by employing a meta-learning paradigm with the objective of improving the performance of masked representations on a typical self-supervised task. We provide solid theoretical analyses to prove MetaMask can obtain tighter risk bounds for downstream classification compared to typical contrastive methods. Empirically, our method achieves state-of-the-art performance on various benchmarks.

Keywords: Representation Learning · Self-supervised learning · contrastive learning · multi-view · dimensional analysis

Author Information

Jiangmeng Li (Institute of Software Chinese Academy of Sciences)
Wenwen Qiang (Institute of Software Chinese Academy of Sciences)
Yanan Zhang (University of the Chinese Academy of Sciences)
Wenyi Mo (Renmin University of China)
Changwen Zheng (Institute of Software, Chinese Academy of Sciences)
Bing Su (Renmin University of China)
Hui Xiong

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors