Timezone: »
In self-supervised learning, a system is tasked with achieving a surrogate objective by defining alternative targets on a set of unlabeled data. The aim is to build useful representations that can be used in downstream tasks, without costly manual annotation. In this work, we propose a novel self-supervised formulation of relational reasoning that allows a learner to bootstrap a signal from information implicit in unlabeled data. Training a relation head to discriminate how entities relate to themselves (intra-reasoning) and other entities (inter-reasoning), results in rich and descriptive representations in the underlying neural network backbone, which can be used in downstream tasks such as classification and image retrieval. We evaluate the proposed method following a rigorous experimental procedure, using standard datasets, protocols, and backbones. Self-supervised relational reasoning outperforms the best competitor in all conditions by an average 14% in accuracy, and the most recent state-of-the-art model by 3%. We link the effectiveness of the method to the maximization of a Bernoulli log-likelihood, which can be considered as a proxy for maximizing the mutual information, resulting in a more efficient objective with respect to the commonly used contrastive losses.
Author Information
Massimiliano Patacchiola (University of Edinburgh)
Massimiliano is a postdoctoral researcher at the University of Cambridge in the Machine Learning Group. He is interested in efficient learning (few-shot, self-supervised, meta-learning), Bayesian methods (Gaussian processes), and reinforcement learning. Previously he has been a postdoctoral researcher at the University of Edinburgh and an intern in the Camera Platform team at Snapchat.
Amos Storkey (University of Edinburgh)
Related Events (a corresponding poster, oral, or spotlight)
-
2020 Spotlight: Self-Supervised Relational Reasoning for Representation Learning »
Thu. Dec 10th 03:00 -- 03:10 PM Room Orals & Spotlights: Unsupervised/Probabilistic
More from the Same Authors
-
2020 : Defining Benchmarks for Continual Few-Shot Learning »
Massimiliano Patacchiola -
2021 : Hamiltonian prior to Disentangle Content and Motion in Image Sequences »
Asif Khan · Amos Storkey -
2022 : Parity in predictive performance is neither necessary nor sufficient for fairness »
Justin Engelmann · Miguel Bernabeu · Amos Storkey -
2022 : Deep Class-Conditional Gaussians for Continual Learning »
Thomas Lee · Amos Storkey -
2022 Poster: Hamiltonian Latent Operators for content and motion disentanglement in image sequences »
Asif Khan · Amos Storkey -
2021 Poster: Memory Efficient Meta-Learning with Large Images »
John Bronskill · Daniela Massiceti · Massimiliano Patacchiola · Katja Hofmann · Sebastian Nowozin · Richard Turner -
2021 Poster: Non-Gaussian Gaussian Processes for Few-Shot Regression »
Marcin Sendera · Jacek Tabor · Aleksandra Nowak · Andrzej Bedychaj · Massimiliano Patacchiola · Tomasz Trzcinski · Przemysław Spurek · Maciej Zieba -
2021 Poster: Gradient-based Hyperparameter Optimization Over Long Horizons »
Paul Micaelli · Amos Storkey -
2020 Poster: Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels »
Massimiliano Patacchiola · Jack Turner · Elliot Crowley · Michael O'Boyle · Amos Storkey -
2020 Spotlight: Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels »
Massimiliano Patacchiola · Jack Turner · Elliot Crowley · Michael O'Boyle · Amos Storkey -
2019 Poster: Zero-shot Knowledge Transfer via Adversarial Belief Matching »
Paul Micaelli · Amos Storkey -
2019 Spotlight: Zero-shot Knowledge Transfer via Adversarial Belief Matching »
Paul Micaelli · Amos Storkey -
2019 Poster: Learning to Learn By Self-Critique »
Antreas Antoniou · Amos Storkey -
2018 Poster: Moonshine: Distilling with Cheap Convolutions »
Elliot Crowley · Gavia Gray · Amos Storkey -
2015 Poster: Covariance-Controlled Adaptive Langevin Thermostat for Large-Scale Bayesian Sampling »
Xiaocheng Shang · Zhanxing Zhu · Benedict Leimkuhler · Amos Storkey -
2014 Workshop: NIPS Workshop on Transactional Machine Learning and E-Commerce »
David Parkes · David H Wolpert · Jennifer Wortman Vaughan · Jacob D Abernethy · Amos Storkey · Mark Reid · Ping Jin · Nihar Bhadresh Shah · Mehryar Mohri · Luis E Ortiz · Robin Hanson · Aaron Roth · Satyen Kale · Sebastien Lahaie -
2012 Poster: Continuous Relaxations for Discrete Hamiltonian Monte Carlo »
Zoubin Ghahramani · Yichuan Zhang · Charles Sutton · Amos Storkey -
2012 Spotlight: Continuous Relaxations for Discrete Hamiltonian Monte Carlo »
Zoubin Ghahramani · Yichuan Zhang · Charles Sutton · Amos Storkey -
2012 Poster: The Coloured Noise Expansion and Parameter Estimation of Diffusion Processes »
Simon Lyons · Amos Storkey · Simo Sarkka -
2011 Poster: Neuronal Adaptation for Sampling-Based Probabilistic Inference in Perceptual Bistability »
David Reichert · Peggy Series · Amos Storkey -
2011 Spotlight: Neuronal Adaptation for Sampling-Based Probabilistic Inference in Perceptual Bistability »
David Reichert · Peggy Series · Amos Storkey -
2010 Poster: Hallucinations in Charles Bonnet Syndrome Induced by Homeostasis: a Deep Boltzmann Machine Model »
David Reichert · Peggy Series · Amos Storkey -
2010 Poster: Sparse Instrumental Variables (SPIV) for Genome-Wide Studies »
Felix V Agakov · Paul McKeigue · Jon Krohn · Amos Storkey -
2007 Poster: Continuous Time Particle Filtering for fMRI »
Lawrence Murray · Amos Storkey -
2007 Poster: Modelling motion primitives and their timing in biologically executed movements »
Ben H Williams · Marc Toussaint · Amos Storkey -
2006 Poster: Learning Structural Equation Models for fMRI »
Amos Storkey · Enrico Simonotto · Heather Whalley · Stephen Lawrie · Lawrence Murray · David McGonigle -
2006 Poster: Mixture Regression for Covariate Shift »
Amos Storkey · Masashi Sugiyama