`

Timezone: »

 
Poster
Dense Associative Memory for Pattern Recognition
Dmitry Krotov · John J. Hopfield

Wed Dec 07 09:00 AM -- 12:30 PM (PST) @ Area 5+6+7+8 #195 #None

A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.

Author Information

Dmitry Krotov (Institute for Advanced Study)
John J. Hopfield (Princeton Neuroscience Institute)

BA Swarthmore 1954; PhD Cornell (theoretical physics) 1958. Member of technical staff Bell Laboratories 1958-1960 & 1973-1996; Faculty positions at UCBerkeley (physics) 1961-1964, Princeton Univ. (physics) 1964-1980, Caltech (chemistry and biology) 1980-1996, Princeton Univ. (molecular biology) 1997-2008, Institute for Advanced Study (2010-2013), now emeritus at Princeton Neuroscience Institute. Served as Chairman of the Faculty, Caltech; President of the American Physical Society; Executive Officer for Computation and Neural Systems, Caltech. Honors include Buckley Prize in Solid State Physics; APS prize in biophysics; Dirac Medal; Einstein Award; MacArthur Fellow; IEEE Rosenblatt Award; Swartz Prize in Computational Neuroscience. Member, National Academy of Science; American Philosophical Society. Research on the interaction of light with solids 1956-1970; biomolecular physics and kinetic proofreading 1970-1980; neural network dynamics and neurobiology 1980-.

More from the Same Authors

  • 2019 : Poster Session »
    Pravish Sainath · Mohamed Akrout · Charles Delahunt · Nathan Kutz · Guangyu Robert Yang · Joseph Marino · L F Abbott · Nicolas Vecoven · Damien Ernst · andrew warrington · Michael Kagan · Kyunghyun Cho · Kameron Harris · Leopold Grinberg · John J. Hopfield · Dmitry Krotov · Taliah Muhammad · Erick Cobos · Edgar Walker · Jacob Reimer · Andreas Tolias · Alexander Ecker · Janaki Sheth · Yu Zhang · Maciej Wołczyk · Jacek Tabor · Szymon Maszke · Roman Pogodin · Dane Corneil · Wulfram Gerstner · Baihan Lin · Guillermo Cecchi · Jenna M Reinen · Irina Rish · Guillaume Bellec · Darjan Salaj · Anand Subramoney · Wolfgang Maass · Yueqi Wang · Ari Pakman · Jin Hyung Lee · Liam Paninski · Bryan Tripp · Colin Graber · Alex Schwing · Luke Prince · Gabriel Ocker · Michael Buice · Benjamin Lansdell · Konrad Kording · Jack Lindsey · Terrence Sejnowski · Matthew Farrell · Eric Shea-Brown · Nicolas Farrugia · Victor Nepveu · Jiwoong Im · Kristin Branson · Brian Hu · Ramakrishnan Iyer · Stefan Mihalas · Sneha Aenugu · Hananel Hazan · Sihui Dai · Tan Nguyen · Doris Tsao · Richard Baraniuk · Anima Anandkumar · Hidenori Tanaka · Aran Nayebi · Stephen Baccus · Surya Ganguli · Dean Pospisil · Eilif Muller · Jeffrey S Cheng · Gaël Varoquaux · Kamalaker Dadi · Dimitrios C Gklezakos · Rajesh PN Rao · Anand Louis · Christos Papadimitriou · Santosh Vempala · Naganand Yadati · Daniel Zdeblick · Daniela M Witten · Nicholas Roberts · Vinay Prabhu · Pierre Bellec · Poornima Ramesh · Jakob H Macke · Santiago Cadena · Guillaume Bellec · Franz Scherr · Owen Marschall · Robert Kim · Hannes Rapp · Marcio Fonseca · Oliver Armitage · Jiwoong Im · Thomas Hardcastle · Abhishek Sharma · Wyeth Bair · Adrian Valente · Shane Shang · Merav Stern · Rutuja Patil · Peter Wang · Sruthi Gorantla · Peter Stratton · Tristan Edwards · Jialin Lu · Martin Ester · Yurii Vlasov · Siavash Golkar
  • 2016 Oral: Dense Associative Memory for Pattern Recognition »
    Dmitry Krotov · John J. Hopfield
  • 2014 Invited Talk: Using the Emergent Dynamics of Attractor Networks for Computation »
    John J. Hopfield