Timezone: »

Modern Hopfield Networks and Attention for Immune Repertoire Classification
Michael Widrich · Bernhard Schäfl · Milena Pavlović · Hubert Ramsauer · Lukas Gruber · Markus Holzleitner · Johannes Brandstetter · Geir Kjetil Sandve · Victor Greiff · Sepp Hochreiter · Günter Klambauer

Wed Dec 09 07:10 AM -- 07:20 AM (PST) @ Orals & Spotlights: COVID/Applications/Composition

A central mechanism in machine learning is to identify, store, and recognize patterns. How to learn, access, and retrieve such patterns is crucial in Hopfield networks and the more recent transformer architectures. We show that the attention mechanism of transformer architectures is actually the update rule of modern Hopfield networks that can store exponentially many patterns. We exploit this high storage capacity of modern Hopfield networks to solve a challenging multiple instance learning (MIL) problem in computational biology: immune repertoire classification. In immune repertoire classification, a vast number of immune receptors are used to predict the immune status of an individual. This constitutes a MIL problem with an unprecedentedly massive number of instances, two orders of magnitude larger than currently considered problems, and with an extremely low witness rate. Accurate and interpretable machine learning methods solving this problem could pave the way towards new vaccines and therapies, which is currently a very relevant research topic intensified by the COVID-19 crisis. In this work, we present our novel method DeepRC that integrates transformer-like attention, or equivalently modern Hopfield networks, into deep learning architectures for massive MIL such as immune repertoire classification. We demonstrate that DeepRC outperforms all other methods with respect to predictive performance on large-scale experiments including simulated and real-world virus infection data and enables the extraction of sequence motifs that are connected to a given disease class. Source code and datasets: https://github.com/ml-jku/DeepRC

Author Information

Michael Widrich (Ellis Unit / University Linz)
Bernhard Schäfl (JKU Linz)
Milena Pavlović (Department of Informatics, University of Oslo)
Hubert Ramsauer (LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria)
Lukas Gruber (Ellis Unit / University Linz)
Markus Holzleitner (Ellis Unit / University Linz)
Johannes Brandstetter (LIT AI Lab / University Linz)
Geir Kjetil Sandve (Department of Informatics, University of Oslo)
Victor Greiff (Department of Immunology, University of Oslo)
Sepp Hochreiter (LIT AI Lab / University Linz / IARAI)
Günter Klambauer (ELLIS Unit University Linz)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors