Timezone: »

 
Poster
Learning Contrastive Embedding in Low-Dimensional Space
Shuo Chen · Chen Gong · Jun Li · Jian Yang · Gang Niu · Masashi Sugiyama

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #710
in Poster Session 1 »

Contrastive learning (CL) pretrains feature embeddings to scatter instances in the feature space so that the training data can be well discriminated. Most existing CL techniques usually encourage learning such feature embeddings in the highdimensional space to maximize the instance discrimination. However, this practice may lead to undesired results where the scattering instances are sparsely distributed in the high-dimensional feature space, making it difficult to capture the underlying similarity between pairwise instances. To this end, we propose a novel framework called contrastive learning with low-dimensional reconstruction (CLLR), which adopts a regularized projection layer to reduce the dimensionality of the feature embedding. In CLLR, we build the sparse / low-rank regularizer to adaptively reconstruct a low-dimensional projection space while preserving the basic objective for instance discrimination, and thus successfully learning contrastive embeddings that alleviate the above issue. Theoretically, we prove a tighter error bound for CLLR; empirically, the superiority of CLLR is demonstrated across multiple domains. Both theoretical and experimental results emphasize the significance of learning low-dimensional contrastive embeddings.

Keywords: Representation Learning · contrastive learning · Autoencoder · dimensionality reduction
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Shuo Chen (RIKEN)
Chen Gong (Nanjing University of Science and Technology)
Jun Li (Nanjing University of Science and Technology)
Jian Yang (Nanjing University of Science and Technology)
Gang Niu (RIKEN)
Gang Niu

Gang Niu is currently an indefinite-term senior research scientist at RIKEN Center for Advanced Intelligence Project.

Masashi Sugiyama (RIKEN / University of Tokyo)

More from the Same Authors