Timezone: »
The macaque Superior Temporal Sulcus (STS) is a brain area that receives and integrates inputs from both the ventral and dorsal visual processing streams (thought to specialize in form and motion processing respectively). For the processing of articulated actions, prior work has shown that even a small population of STS neurons contains sufficient information for the decoding of actor invariant to action, action invariant to actor, as well as the specific conjunction of actor and action. This paper addresses two questions. First, what are the invariance properties of individual neural representations (rather than the population representation) in STS? Second, what are the neural encoding mechanisms that can produce such individual neural representations from streams of pixel images? We find that a baseline model, one that simply computes a linear weighted sum of ventral and dorsal responses to short action “snippets”, produces surprisingly good fits to the neural data. Interestingly, even using inputs from a single stream, both actor-invariance and action-invariance can be produced simply by having different linear weights.
Author Information
Cheston Tan (Institute for Infocomm Research, Singapore)
Jedediah M Singer (Boston Children's Hospital)
Thomas Serre (Brown University)
David Sheinberg (Brown University)
Tomaso Poggio (MIT)
Tomaso A. Poggio, is the Eugene McDermott Professor in the Dept. of Brain & Cognitive Sciences at MIT and the director of the new NSF Center for Brains, Minds and Machines at MIT of which MIT and Harvard are the main member Institutions. He is a member of both the Computer Science and Artificial Intelligence Laboratory and of the McGovern Brain Institute. He is an honorary member of the Neuroscience Research Program, a member of the American Academy of Arts and Sciences, a Founding Fellow of AAAI and a founding member of the McGovern Institute for Brain Research. Among other honors he received the Laurea Honoris Causa from the University of Pavia for the Volta Bicentennial, the 2003 Gabor Award, the Okawa Prize 2009, the AAAS Fellowship and the 2014 Swartz Prize for Theoretical and Computational Neuroscience. He is one of the most cited computational scientists with contributions ranging from the biophysical and behavioral studies of the visual system to the computational analyses of vision and learning in humans and machines. With W. Reichardt he characterized quantitatively the visuo-motor control system in the fly. With D. Marr, he introduced the seminal idea of levels of analysis in computational neuroscience. He introduced regularization as a mathematical framework to approach the ill-posed problems of vision and the key problem of learning from data. In the last decade he has developed an influential hierarchical model of visual recognition in the visual cortex. The citation for the recent 2009 Okawa prize mentions his ââ¦outstanding contributions to the establishment of computational neuroscience, and pioneering researches ranging from the biophysical and behavioral studies of the visual system to the computational analysis of vision and learning in humans and machines.â His research has always been interdisciplinary, between brains and computers. It is now focused on the mathematics of learning theory, the applications of learning techniques to computer vision and especially on computational neuroscience of the visual cortex. A former Corporate Fellow of Thinking Machines Corporation and a former director of PHZ Capital Partners, Inc., he is a director of Mobileye and was involved in starting, or investing in, several other high tech companies including Arris Pharmaceutical, nFX, Imagen, Digital Persona and Deep Mind. Tomaso Poggio Eugene McDermott Professor Director NSF Science & Technology Center for Brains, Minds and Machines(CBMM) http://cbmm.mit.edu/ Core founding scientific advisor, MIT Quest for Intelligence McGovern Institute CSAIL (Computer Science and Artificial Intelligence Lab) Brain Sciences Department M.I.T., 46-5177B see http://whereis.mit.edu/?selection=46&Buildings=go 43 Vassar Street Cambridge, MA 02142 E-mail: tp@ai.mit.edu Phone: 617-253-5230 Fax: 617-253-2964 Web: http://cbcl.mit.edu/people/poggio/poggio-new.htm PoggioLab Web page: http://cbcl.mit.edu/
More from the Same Authors
-
2022 : The emergence of visual simulation in task-optimized recurrent neural networks »
Alekh Karkada Ashok · Lakshmi Narasimhan Govindarajan · Drew Linsley · David Sheinberg · Thomas Serre -
2023 Poster: Break It Down: Evidence for Structural Compositionality in Neural Networks »
Michael Lepori · Thomas Serre · Ellie Pavlick -
2023 Poster: Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex »
Drew Linsley · Ivan F Rodriguez Rodriguez · Thomas FEL · Michael Arcaro · Saloni Sharma · Margaret Livingstone · Thomas Serre -
2023 Poster: A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation »
Thomas FEL · Victor Boutin · Louis Béthune · Remi Cadene · Mazda Moayeri · Léo Andéol · Mathieu Chalvidal · Thomas Serre -
2023 Poster: Unlocking Feature Visualization for Deep Network with MAgnitude Constrained Optimization »
Thomas FEL · Thibaut Boissin · Victor Boutin · Agustin PICARD · Paul Novello · Julien Colin · Drew Linsley · Tom ROUSSEAU · Remi Cadene · Laurent Gardes · Thomas Serre -
2023 Poster: Learning Functional Transduction »
Mathieu Chalvidal · Thomas Serre · Rufin VanRullen -
2023 Poster: Computing a human-like reaction time metric from stable recurrent vision models »
Lore Goetschalckx · Lakshmi Narasimhan Govindarajan · Alekh Karkada Ashok · Thomas Serre -
2022 Poster: Meta-Reinforcement Learning with Self-Modifying Networks »
Mathieu Chalvidal · Thomas Serre · Rufin VanRullen -
2022 Poster: A Benchmark for Compositional Visual Reasoning »
Aimen Zerroug · Mohit Vaishnav · Julien Colin · Sebastian Musslick · Thomas Serre -
2022 Poster: Diversity vs. Recognizability: Human-like generalization in one-shot generative models »
Victor Boutin · Lakshya Singhal · Xavier Thomas · Thomas Serre -
2022 Poster: Harmonizing the object recognition strategies of deep neural networks with humans »
Thomas FEL · Ivan F Rodriguez Rodriguez · Drew Linsley · Thomas Serre -
2022 Poster: What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods »
Julien Colin · Thomas FEL · Remi Cadene · Thomas Serre -
2021 : AVoE: A Synthetic 3D Dataset on Understanding Violation of Expectation for Artificial Cognition »
Arijit Dasgupta · Jiafei Duan · Marcelo Ang Jr · Cheston Tan -
2021 Poster: Tracking Without Re-recognition in Humans and Machines »
Drew Linsley · Girik Malik · Junkyung Kim · Lakshmi Narasimhan Govindarajan · Ennio Mingolla · Thomas Serre -
2021 Poster: Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis »
Thomas FEL · Remi Cadene · Mathieu Chalvidal · Matthieu Cord · David Vigouroux · Thomas Serre -
2021 Poster: Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee »
Xiaofeng Fan · Yining Ma · Zhongxiang Dai · Wei Jing · Cheston Tan · Bryan Kian Hsiang Low -
2020 Poster: Stable and expressive recurrent vision models »
Drew Linsley · Alekh Karkada Ashok · Lakshmi Narasimhan Govindarajan · Rex Liu · Thomas Serre -
2020 Poster: Biologically Inspired Mechanisms for Adversarial Robustness »
Manish Reddy Vuyyuru · Andrzej Banburski · Nishka Pant · Tomaso Poggio -
2020 Spotlight: Stable and expressive recurrent vision models »
Drew Linsley · Alekh Karkada Ashok · Lakshmi Narasimhan Govindarajan · Rex Liu · Thomas Serre -
2020 Session: Orals & Spotlights Track 29: Neuroscience »
Aasa Feragen · Thomas Serre -
2018 Poster: Learning long-range spatial dependencies with horizontal gated recurrent units »
Drew Linsley · Junkyung Kim · Vijay Veerabadran · Charles Windolf · Thomas Serre -
2017 Poster: Do Deep Neural Networks Suffer from Crowding? »
Anna Volokitin · Gemma Roig · Tomaso Poggio -
2016 Poster: How Deep is the Feature Analysis underlying Rapid Visual Categorization? »
Sven Eberhardt · Jonah G Cader · Thomas Serre -
2015 Symposium: Brains, Minds and Machines »
Gabriel Kreiman · Tomaso Poggio · Maximilian Nickel -
2015 Poster: Learning with Group Invariant Features: A Kernel Perspective. »
Youssef Mroueh · Stephen Voinea · Tomaso Poggio -
2015 Poster: Learning with a Wasserstein Loss »
Charlie Frogner · Chiyuan Zhang · Hossein Mobahi · Mauricio Araya · Tomaso Poggio -
2012 Poster: Learning Manifolds with K-Means and K-Flats »
Guillermo D Canas · Tomaso Poggio · Lorenzo Rosasco -
2012 Poster: Multiclass Learning with Simplex Coding »
Youssef Mroueh · Tomaso Poggio · Lorenzo Rosasco · Jean-Jacques Slotine -
2009 Poster: On Invariance in Hierarchical Models »
Jake Bouvrie · Lorenzo Rosasco · Tomaso Poggio -
2008 Workshop: Cortical Microcircuits and their Computational Functions »
Tomaso Poggio · Terrence Sejnowski -
2007 Tutorial: Visual Recognition in Primates and Machines »
Tomaso Poggio