Optimization Principles in Neural Coding and Computation
Abstract: We walk through the world largely unaware of the computations done by our brains. Although we tend to focus on our errors, most of the time these computations seem to get the right answer or we probably wouldn't be here to argue about it. But is this process one in which a complicated evolutionary history has led to messy solutions that are nonetheless "good enough" for survival, or is it possible that the brain in fact finds optimal or near optimal solutions to its problems? Optimal solutions would be attractive, not least because this would open the way to having a real theory of neural computation-based optimization principles, in effect a theory of why the nervous system works the way it does rather than just a model of how it works; this is an ambitious goal. In this tutorial I will review the evidence for optimal performance at a wide variety of tasks in neural coding and computation, ranging from photon counting in vision to echolocation in bats and pitch perception in human hearing. Recent work has focused attention on the crucial prediction that -- given the statistical structure of our sensory world -- optimal strategies are almost always adaptive, and hence one can do experiments to observe the process of optimization in action; this has led to some dramatic results, providing strong support for the applicability of optimization principles. I will try to emphasize the generality of this approach, looking at many different sensory systems and also making connections to work on the motor system described in Wolpert's tutorial at this meeting. I will address some classic examples of superficially sub-optimal behavior, and argue that optimization principles for neural computation can generate counterintuitive predictions that may illuminate these paradoxes. Finally, we'll step back and ask if the many different optimization ideas can be unified, and if so whether this provides a glimpse of a theory that is strong enough to carry us from low level sensory processing all the way to cognition.
Bio: William Bialek is the John Archibald Wheeler/Battelle Professor in Physics at Princeton University. He received his PhD in Biophysics from the University of California, Berkeley. After postdoctoral appointments at the Rijksuniversiteit Groningen in the Netherlands and at the Institute for Theoretical Physics in Santa Barbara, he returned to Berkeley to join the faculty in 1986. In late 1990, he moved to the newly formed NEC Research Institute, where he eventually became an Institute Fellow. He joined the Princeton faculty in 2001. Best known for contributions to our understanding of neural coding, Bialek and his collaborators have worked on a wide variety of theoretical problems at the interface of physics and biology, from the dynamics of individual biological molecules to learning and cognition. He served as co-director of the computational neuroscience course at the Marine Biological Laboratory in Woods Hole, Massachusetts from 1998 to 2002, and currently is involved in a major educational experiment at Princeton to create an integrated and mathematically sophisticated introduction to the natural sciences for first year college students.
Tutorial: A Geometric Approach to Statistical Learning Theory
Abstract: We will present a survey of the recent developments in Statistical Learning Theory from a geometric point of view. It turns out that understanding the "hardness" of a prediction problem is governed by the geometry of the class of functions one uses; in particular, what is most important is the richness of finite dimensional coordinate projection the class (that is, the restriction of the class to the sample points) as the dimension of the projection (the size of the sample) increases. This observation has resulted in a fruitful interplay between statistical learning theory and asymptotic geometry. We will focus on two aspects which were particularly influenced by Asymptotic Geometry. The first is prediction bounds. We will show how the complexity of the class, as captured by the geometry of its coordinate projections, determines the error rate, and the way classical parameters, such as the combinatorial dimension (a real-valued version of the VC dimension) or the covering numbers are simply a sufficient condition which ensures the required control on the geometry of the class. The second topic we will explore is the limitation of kernel methods. For example, we will present recent results which show that kernel methods, or even any conceivable extension of these methods (embedding into other spaces, other choices of a loss functional, etc), are bound to fail in a very strong sense.
Bio: Shahar Mendelson is a Fellow at the Institute of Advanced Studies at the Australian National University. He received his BA in Mathematics in 1992, MSc in Mathematics in 1994 and PhD in Applied Mathematics in 1998, all from the Technion, I.I.T (Haifa, Israel). His research interest is high dimensional phenomena, and in particular, the connections between Asymptotic Geometric Analysis and Learning Theory.
Tutorial: Bayesian Methods for Machine Learning
Abstract: The Bayesian approach to machine learning is based on the idea that all forms of uncertainty should be represented using probability. This begins with a formulation of the problem in terms of a probabilistic model along with a "prior'' probability distribution for the model's parameters, and continues through to the production of predictions in probabilistic form. The Bayesian approach differs fundamentally from "learning theory'' approaches, which focus on concepts such as VC dimension and the bias-variance tradeoff, in which probability plays a more restricted role. This difference mirrors the long-standing contention in statistics between Bayesian and "frequentist'' approaches to inference.
In this tutorial, I will introduce the Bayesian idea with some simple examples, and show how it leads to some fundamental differences in how one approaches problems in machine learning. I will then discuss the two big challenges in applying Bayesian methods in practice - how to deal intellectually with the complexity of realistic models, and how to deal computationally with the high-dimensional integrals that are needed to produce Bayesian predictions. As examples of complex models that can nevertheless be understood well enough to be useful, I will briefly discuss Bayesian neural network models, Gaussian process models, and Bayesian mixture models with an infinite number of components. I will also briefly discuss Markov chain Monte Carlo and variational methods for performing Bayesian computations. I will then present some illustrations of how the Bayesian approach can be used to tackle complex problems that are difficult to handle by other methods, and to achieve performance on simpler problems that is superior to that of non-Bayesian methods. Finally, I will discuss the current limitations of Bayesian methods, as well as some currently popular Bayesian techniques that need to be used with caution.
Bio: Radford Neal is a professor in the departments of Statistics and Computer Science at the University of Toronto, where he holds the Canada Research Chair in Statistics and Machine Learning. He received his BSc and MSc degrees in Computer Science from the University of Calgary in 1977 and 1980, after which he spent some years working on software development and research projects in industry and academia. He received his PhD in Computer Science from the University of Toronto in 1994, and then joined the University of Toronto faculty. His primary research interests are flexible Bayesian models for complex problems, computation for such models using Markov chain Monte Carlo methods, and applications of such models, especially to problems in bioinformatics. He is the first recipient of the Lindley Prize, awarded for innovative work in Bayesian statistics appearing as a contributed paper in the proceedings of the Valencia and ISBA conferences.
Computational Mechanism Design and Auctions
Abstract: Computational Mechanism Design (CMD) aims to develop protocols to implement desirable outcomes in a distributed system of self-interested agents. An outcome might define a joint plan of action, or an allocation of resources to agents. CMD brings together economic concepts of game-theoretic equilibrium and incentive-compatibility constraints with the traditional computer science concerns of computational tractability and communication complexity.
This tutorial is designed for a computer science audience, and it should have interest to both novices and experts in computational mechanism design and computational game theory. As motivation, we will begin with a brief survey of some of the problems in multi-agent systems to which CMD can be applied. The introduction will also provide a brief review of the most relevant solution concepts from game theory. The tutorial will close with some extended comments about future directions in CMD. The bulk of the tutorial will be devoted to the following four topics: i) Existing characterizations of strategyproof mechanisms; ii) The Vickrey-Clarke-Groves mechanism: A case-study in centralized implementation and the tension between approximation and strategyproofness. iii) Iterative mechanisms: Towards decentralized implementations (addressing communication and valuation complexity). iv) Future directions: Sequential implementation; Distributed implementation; and Adaptive implementation.
Bio: David Parkes is a Gordon McKay Assistant Professor of Computer Science at Harvard University. He received his Ph.D. degree in Computer and Information Science from the University of Pennsylvania in 2001, and an M.Eng. (First class) in Engineering and Computing Science from Oxford University in 1995. He has published technical papers on electronic commerce, auction design, computational mechanism design, multi-agent systems, and bounded-rationality. Dr. Parkes was awarded the prestigious NSF CAREER Award in 2002, and the IBM Faculty Partnership Award in 2002 and 2003. He serves on the editorial board of the Journal of Artificial Intelligence Research, and on the Program Committee of a number of leading conferences in artificial intelligence, multiagent systems, and electronic commerce.
Acquiring Detailed 3D Models From Images and Video
Abstract: Creating photo-realistic 3D models from images and video has been one of the long-standing goals of computer vision. More recently, image-based rendering techniques have been developed in computer graphics for manipulating and rendering such models while maintaining visual fidelity, often by going back to the original data and using various forms of sampling. This parallels a similar evolution in many other fields, including speech recognition and AI, where data-driven and learning approaches now predominate over what used to be hand-crafted models and algorithms.
In this tutorial, I will overview how 3D and temporal models are constructed from images and video. This includes feature matching, camera pose estimation, structure (shape) recovery, and appearance (albedo and BRDF) estimation. In the temporal domain, I will discuss how Video Textures and other video-based rendering techniques can maintain visual realism using data-driven approaches. I will also discuss open research problems where machine learning and statistical approaches show promise for future improvements.
Bio: Richard Szeliski is a Senior Researcher in the Vision Technology Group at Microsoft Research, where he is pursuing research in 3-D computer vision, video scene analysis, and image-based rendering. His current focus is on constructing photorealistic 3D scene models from multiple images and video. He received a Ph.D. degree in Computer Science from Carnegie Mellon University, Pittsburgh, in 1988. He joined Microsoft Research in 1995. Prior to Microsoft, he worked at Bell-Northern Research, Schlumberger Palo Alto Research, the Artificial Intelligence Center of SRI International, and the Cambridge Research Lab of Digital Equipment Corporation. Dr. Szeliski has published over 100 research papers in computer vision, computer graphics, medical imaging, and neural nets, as well as the book Bayesian Modeling of Uncertainty in Low-Level Vision. He was a Program Committee Chair for ICCV'2001, and has served on the Editorial Board of the International Journal of Computer Vision and as an Associate Editor of the IEEE Transactions on Pattern Analysis and Machine Intelligence.
Probabilistic Computations in Human Sensorimotor Control
Abstract: Although computers can now beat grandmasters at chess, no robot can manipulate a chess piece with the dexterity of a six-year-old child. In this tutorial I will explore techniques for reverse engineering the human sensorimotor control system. Over recent years there has been an explosion of interest in using machine learning algorithms to understand human performance. This tutorial will first provide a brief overview of the new experimental methodologies available to study human sensorimotor control, such as robotic and virtual reality interfaces. In the second part I will review how machine learning concepts can be tested experimentally in humans. This part will cover approaches such as Bayesian decision theory, reinforcement learning, representation and generalization, optimal control and learning, state estimation and prediction. This tutorial is well-suited to theorists who would like to learn how to test their models or develop algorithms that are inspired by human performance, as well as to experimentalist who wish for a stronger link to theory.
Bio: Daniel Wolpert is Professor of Motor Neuroscience and co-director of the Institute of Movement Neuroscience at the Institute of Neurology, University College London. He received his bachelor's degree in medical sciences at Cambridge University in 1985 and a clinical medical degree from Oxford University in 1988. He received a PhD in Physiology at Oxford University in 1992 and subsequently worked as a postdoctoral fellow in Michael Jordan's group in the Department of Brain and Cognitive Sciences at MIT. He joined the Sobell Department of Motor Neuroscience & Movement Disorders, Institute of Neurology in 1995. His research interests are computational and experimental approaches to human sensorimotor control.