Timezone: »
Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models. Modern batch contrastive approaches subsume or significantly outperform traditional contrastive losses such as triplet, max-margin and the N-pairs loss. In this work, we extend the self-supervised batch contrastive approach to the fully-supervised setting, allowing us to effectively leverage label information. Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes. We analyze two possible versions of the supervised contrastive (SupCon) loss, identifying the best-performing formulation of the loss. On ResNet-200, we achieve top-1 accuracy of 81.4% on the ImageNet dataset, which is 0.8% above the best number reported for this architecture. We show consistent outperformance over cross-entropy on other datasets and two ResNet variants. The loss shows benefits for robustness to natural corruptions, and is more stable to hyperparameter settings such as optimizers and data augmentations. In reduced data settings, it outperforms cross-entropy significantly. Our loss function is simple to implement and reference TensorFlow code is released at https://t.ly/supcon.
Author Information
Prannay Khosla (Google LLC)
Piotr Teterwak (Google)
Chen Wang (Google)
Aaron Sarna (Google)
Yonglong Tian (MIT)
Phillip Isola (Massachusetts Institute of Technology)
Aaron Maschinot (Google Research)
Ce Liu (Google)
Dilip Krishnan (Google)
More from the Same Authors
-
2021 : The Neural MMO Platform for Massively Multiagent Research »
Joseph Suarez · Yilun Du · Clare Zhu · Igor Mordatch · Phillip Isola -
2021 Spotlight: Learning to See by Looking at Noise »
Manel Baradad Jurjo · Jonas Wulff · Tongzhou Wang · Phillip Isola · Antonio Torralba -
2022 : Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings »
Tongzhou Wang · Phillip Isola -
2022 : Real world relevance of generative counterfactual explanations »
Swami Sankaranarayanan · Thomas Hartvigsen · Lauren Oakden-Rayner · Marzyeh Ghassemi · Phillip Isola -
2022 : Fifteen-minute Competition Overview Video »
Kate Saenko · Samarth Mishra · Dina Bashkirova · Vitaly Ablavsky · Sarah Bargal · Rachel Lai · Piotr Teterwak · James Akl · Fadi Alladkani · Donghyun Kim · Berk Calli -
2023 Poster: StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners »
Yonglong Tian · Lijie Fan · Phillip Isola · Huiwen Chang · Dilip Krishnan -
2023 Poster: Improving CLIP Training with Language Rewrites »
Lijie Fan · Dilip Krishnan · Phillip Isola · Dina Katabi · Yonglong Tian -
2023 Poster: Learning New Dimensions of Human Visual Similarity using Synthetic Data »
Stephanie Fu · Netanel Tamir · Shobhita Sundaram · Lucy Chai · Richard Zhang · Tali Dekel · Phillip Isola -
2023 Poster: Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning »
Joseph Suarez · David Bloomin · Kyoung Choe · Hao Xiang Li · Ryan Sullivan · Nishaanth Kanna · Daniel Scott · Rose Shuman · Herbie Bradley · Louis Castricato · Phillip Isola · Chenghui Yu · Yuhao Jiang · Qimai Li · Jiaxin Chen · Xiaolong Zhu -
2023 Competition: The NeurIPS 2023 Neural MMO Challenge: Multi-Task Reinforcement Learning and Curriculum Generation »
Joseph Suarez · Phillip Isola · David Bloomin · Kyoung Choe · Hao Li · Ryan Sullivan · Nishaanth Kanna · Daniel Scott · Rose Shuman · Herbie Bradley · Louis Castricato · Chenghui Yu · Yuhao Jiang · Qimai Li · Jiaxin Chen · Xiaolong Zhu · Dipam Chakrabroty · Sharada Mohanty -
2022 : Final Q&A and Discussion Session »
Ian Goodine · Sujit Sanjeev · Amanda Marrs · Subhransu Maji · Colorado Reed · Binhui Xie · Dong-Geol Choi · Shahaf Ettedgui · Dina Bashkirova · Samarth Mishra · Piotr Teterwak · Donghyun Kim · Diala Lteif -
2022 Competition: The Third Neural MMO Challenge: Learning to Specialize in Massively Multiagent Open Worlds »
Joseph Suarez · Hanmo Chen · Arbin Chen · Bo Wu · Xiaolong Zhu · enhong liu · JUN HU · Chenghui Yu · Phillip Isola -
2022 Competition: VisDA 2022 Challenge: Sim2Real Domain Adaptation for Industrial Recycling »
Dina Bashkirova · Samarth Mishra · Piotr Teterwak · Donghyun Kim · Rachel Lai · Fadi Alladkani · James Akl · Vitaly Ablavsky · Sarah Bargal · Berk Calli · Kate Saenko -
2022 : Challenge Introduction »
Dina Bashkirova · Samarth Mishra · Piotr Teterwak · Donghyun Kim · Sarah Bargal · Diala Lteif · Kate Saenko -
2022 : Line Drawings as Communication »
Phillip Isola -
2022 Poster: Semantic uncertainty intervals for disentangled latent spaces »
Swami Sankaranarayanan · Anastasios Angelopoulos · Stephen Bates · Yaniv Romano · Phillip Isola -
2022 Poster: Procedural Image Programs for Representation Learning »
Manel Baradad · Richard Chen · Jonas Wulff · Tongzhou Wang · Rogerio Feris · Antonio Torralba · Phillip Isola -
2022 Poster: Offline Multi-Agent Reinforcement Learning with Knowledge Distillation »
Wei-Cheng Tseng · Tsun-Hsuan Johnson Wang · Yen-Chen Lin · Phillip Isola -
2021 Poster: Learning to Ground Multi-Agent Communication with Autoencoders »
Toru Lin · Jacob Huh · Christopher Stauffer · Ser Nam Lim · Phillip Isola -
2021 Poster: Learning to See by Looking at Noise »
Manel Baradad Jurjo · Jonas Wulff · Tongzhou Wang · Phillip Isola · Antonio Torralba -
2021 Poster: Adaptable Agent Populations via a Generative Model of Policies »
Kenneth Derek · Phillip Isola -
2021 : VisDA21: Visual Domain Adaptation + Q&A »
Kate Saenko · Kuniaki Saito · Donghyun Kim · Samarth Mishra · Ben Usman · Piotr Teterwak · Dina Bashkirova · Dan Hendrycks -
2020 Poster: What Makes for Good Views for Contrastive Learning? »
Yonglong Tian · Chen Sun · Ben Poole · Dilip Krishnan · Cordelia Schmid · Phillip Isola -
2020 Session: Orals & Spotlights Track 07: Vision Applications »
Ce Liu · Natalia Neverova -
2020 : Discussion Panel: Hugo Larochelle, Finale Doshi-Velez, Devi Parikh, Marc Deisenroth, Julien Mairal, Katja Hofmann, Phillip Isola, and Michael Bowling »
Hugo Larochelle · Finale Doshi-Velez · Marc Deisenroth · Devi Parikh · Julien Mairal · Katja Hofmann · Phillip Isola · Michael Bowling -
2019 : Contributed Session - Spotlight Talks »
Jonathan Frankle · David Schwab · Ari Morcos · Qianli Ma · Yao-Hung Hubert Tsai · Ruslan Salakhutdinov · YiDing Jiang · Dilip Krishnan · Hossein Mobahi · Samy Bengio · Sho Yaida · Muqiao Yang -
2019 : Poster Presentations »
Rahul Mehta · Andrew Lampinen · Binghong Chen · Sergio Pascual-Diaz · Jordi Grau-Moya · Aldo Faisal · Jonathan Tompson · Yiren Lu · Khimya Khetarpal · Martin Klissarov · Pierre-Luc Bacon · Doina Precup · Thanard Kurutach · Aviv Tamar · Pieter Abbeel · Jinke He · Maximilian Igl · Shimon Whiteson · Wendelin Boehmer · Raphaël Marinier · Olivier Pietquin · Karol Hausman · Sergey Levine · Chelsea Finn · Tianhe Yu · Lisa Lee · Benjamin Eysenbach · Emilio Parisotto · Eric Xing · Ruslan Salakhutdinov · Hongyu Ren · Anima Anandkumar · Deepak Pathak · Christopher Lu · Trevor Darrell · Alexei Efros · Phillip Isola · Feng Liu · Bo Han · Gang Niu · Masashi Sugiyama · Saurabh Kumar · Janith Petangoda · Johan Ferret · James McClelland · Kara Liu · Animesh Garg · Robert Lange -
2019 : Oral Presentations »
Janith Petangoda · Sergio Pascual-Diaz · Jordi Grau-Moya · Raphaël Marinier · Olivier Pietquin · Alexei Efros · Phillip Isola · Trevor Darrell · Christopher Lu · Deepak Pathak · Johan Ferret -
2019 : Lunch Break and Posters »
Xingyou Song · Elad Hoffer · Wei-Cheng Chang · Jeremy Cohen · Jyoti Islam · Yaniv Blumenfeld · Andreas Madsen · Jonathan Frankle · Sebastian Goldt · Satrajit Chatterjee · Abhishek Panigrahi · Alex Renda · Brian Bartoldson · Israel Birhane · Aristide Baratin · Niladri Chatterji · Roman Novak · Jessica Forde · YiDing Jiang · Yilun Du · Linara Adilova · Michael Kamp · Berry Weinstein · Itay Hubara · Tal Ben-Nun · Torsten Hoefler · Daniel Soudry · Hsiang-Fu Yu · Kai Zhong · Yiming Yang · Inderjit Dhillon · Jaime Carbonell · Yanqing Zhang · Dar Gilboa · Johannes Brandstetter · Alexander R Johansen · Gintare Karolina Dziugaite · Raghav Somani · Ari Morcos · Freddie Kalaitzis · Hanie Sedghi · Lechao Xiao · John Zech · Muqiao Yang · Simran Kaur · Qianli Ma · Yao-Hung Hubert Tsai · Ruslan Salakhutdinov · Sho Yaida · Zachary Lipton · Daniel Roy · Michael Carbin · Florent Krzakala · Lenka Zdeborová · Guy Gur-Ari · Ethan Dyer · Dilip Krishnan · Hossein Mobahi · Samy Bengio · Behnam Neyshabur · Praneeth Netrapalli · Kris Sankaran · Julien Cornebise · Yoshua Bengio · Vincent Michalski · Samira Ebrahimi Kahou · Md Rifat Arefin · Jiri Hron · Jaehoon Lee · Jascha Sohl-Dickstein · Samuel Schoenholz · David Schwab · Dongyu Li · Sang Choe · Henning Petzka · Ashish Verma · Zhichao Lin · Cristian Sminchisescu -
2019 Poster: Adversarial Robustness through Local Linearization »
Chongli Qin · James Martens · Sven Gowal · Dilip Krishnan · Krishnamurthy Dvijotham · Alhussein Fawzi · Soham De · Robert Stanforth · Pushmeet Kohli -
2019 Poster: Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity »
Deepak Pathak · Christopher Lu · Trevor Darrell · Phillip Isola · Alexei Efros -
2019 Spotlight: Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity »
Deepak Pathak · Christopher Lu · Trevor Darrell · Phillip Isola · Alexei Efros -
2018 Poster: Large Margin Deep Networks for Classification »
Gamaleldin Elsayed · Dilip Krishnan · Hossein Mobahi · Kevin Regan · Samy Bengio -
2016 Poster: Domain Separation Networks »
Konstantinos Bousmalis · George Trigeorgis · Nathan Silberman · Dilip Krishnan · Dumitru Erhan -
2011 Poster: Understanding the Intrinsic Memorability of Images »
Phillip Isola · Devi Parikh · Antonio Torralba · Aude Oliva