Timezone: »

 
Invited Talk
Deep Learning with Multiplicative Interactions
Geoffrey E Hinton

Tue Dec 08 02:00 PM -- 03:00 PM (PST) @
in Oral Session 3: Deep Learning and Network Models »

Deep networks can be learned efficiently from unlabeled data. The layers of representation are learned one at a time using a simple learning module that has only one layer of latent variables. The values of the latent variables of one module form the data for training the next module. The most commonly used modules are Restricted Boltzmann Machines or autoencoders with a sparsity penalty on the hidden activities. Although deep networks have been quite successful for tasks such as object recognition, information retrieval, and modeling motion capture data, the simple learning modules do not have multiplicative interactions which are very useful for some types of data.

The talk will show how a third-order energy function can be factorized to yield a simple learning module that retains advantageous properties of a Restricted Boltzmann Machine such as very simple exact inference and a very simple learning rule based on pair-wise statistics. The new module contains multiplicative interactions that are useful for a variety of unsupervised learning tasks. Researchers at the University of Toronto have been using this type of module to extract oriented energy from image patches and dense flow fields from image sequences. The new module can also be used to allow the style of a motion to blend autoregressive models of motion capture data. Finally, the new module can be used to combine an eye-position with a feature-vector to allow a system that has a variable resolution retina to integrate information about shape over many fixations.

Author Information

Geoffrey E Hinton (Google & University of Toronto)

Geoffrey Hinton received his PhD in Artificial Intelligence from Edinburgh in 1978 and spent five years as a faculty member at Carnegie-Mellon where he pioneered back-propagation, Boltzmann machines and distributed representations of words. In 1987 he became a fellow of the Canadian Institute for Advanced Research and moved to the University of Toronto. In 1998 he founded the Gatsby Computational Neuroscience Unit at University College London, returning to the University of Toronto in 2001. His group at the University of Toronto then used deep learning to change the way speech recognition and object recognition are done. He currently splits his time between the University of Toronto and Google. In 2010 he received the NSERC Herzberg Gold Medal, Canada's top award in Science and Engineering.

More from the Same Authors