University of Toronto
Deep Belief Nets
3:30 - 5:30pm Monday, December 03, 2007
Complex probabilistic models of unlabeled data can be created by combining simpler models. Mixture models are obtained by averaging the densities of simpler models and "products of experts" are obtained by multiplying the densities together and renormalizing. A far more powerful type of combination is to form a "composition of experts" by treating the values of the latent variables of one model as the data for learning the next model. The first half of the tutorial will show how deep belief nets -- directed generative models with many layers of hidden variables -- can be learned one layer at a time by composing simple, undirected, product of expert models that only have one hidden layer. It will also explain why composing directed models does not work. Deep belief nets are trained as generative models on large, unlabeled datasets, but once multiple layers of features have been created by unsupervised learning, they can be fine-tuned to give excellent discrimination on small, labeled datasets. The second half of the tutorial will describe applications of deep belief nets to several tasks including object recognition, non-linear dimensionality reduction, document retrieval, and the interpretation of medical images. It will also show how the learning procedure for deep belief nets can be extended to high-dimensional time series and hierarchies of Conditional Random Fields.
Geoffrey Hinton received his PhD in Artificial Intelligence from Edinburgh in 1978. He did postdoctoral work at the University of California San Diego and spent five years as a faculty member in the Computer Science Department at Carnegie-Mellon. He then became a fellow of the Canadian Institute for Advanced Research and moved to the Department of Computer Science at the University of Toronto. He spent three years from 1998 until 2001 setting up the Gatsby Computational Neuroscience Unit at University College London and then returned to the University of Toronto where he is the Raymond Reiter Distinguished Professor of Artificial Intelligence. He is a fellow of the Royal Society and an honorary foreign member of the American Academy of Arts. He has been awarded the Rumelhart prize and the Research Excellence award of the International Joint Conference on Artificial Intelligence. Geroffrey Hinton was one of the researchers who introduced the back-propagation algorithm that has been widely used for practical applications. His other main contributions to neural network research include Boltzmann machines, distributed representations, time-delay neural nets, mixtures of experts , Helmholtz machines, products of experts, and deep learning on unlabeled data by stacking simple learning modules.
Slides (PowerPoint)
Flash Movie Session A