Timezone: »
A fundamental goal of systems neuroscience is to understand the relationship between neural activity and behavior. Behavior has traditionally been characterized by low-dimensional, task-related variables such as movement speed or response times. More recently, there has been a growing interest in automated analysis of high-dimensional video data collected during experiments. Here we introduce a probabilistic framework for the analysis of behavioral video and neural activity. This framework provides tools for compression, segmentation, generation, and decoding of behavioral videos. Compression is performed using a convolutional autoencoder (CAE), which yields a low-dimensional continuous representation of behavior. We then use an autoregressive hidden Markov model (ARHMM) to segment the CAE representation into discrete "behavioral syllables." The resulting generative model can be used to simulate behavioral video data. Finally, based on this generative model, we develop a novel Bayesian decoding approach that takes in neural activity and outputs probabilistic estimates of the full-resolution behavioral video. We demonstrate this framework on two different experimental paradigms using distinct behavioral and neural recording technologies.
Author Information
Eleanor Batty (Columbia University)
Matthew Whiteway (Columbia University)
Shreya Saxena (Columbia University)
Dan Biderman (Columbia University)
Taiga Abe (Columbia University)
Simon Musall (Cold Spring Harbor Laboratory)
Winthrop Gillis (Harvard Medical School)
Jeffrey Markowitz (Harvard Medical School)
Anne Churchland (Cold Spring Harbor Laboratory)
John Cunningham (University of Columbia)
Sandeep R Datta (Harvard Medical School)
Scott Linderman (Stanford University)
Liam Paninski (Columbia University)
More from the Same Authors
-
2022 : The Best Deep Ensembles Sacrifice Predictive Diversity »
Taiga Abe · Estefany Kelly Buchanan · Geoff Pleiss · John Cunningham -
2022 : Denoising Deep Generative Models »
Gabriel Loaiza-Ganem · Brendan Ross · Luhuan Wu · John Cunningham · Jesse Cresswell · Anthony Caterini -
2022 : Neural encoding and decoding of facial movements »
Scott Linderman -
2022 Poster: Data Augmentation for Compositional Data: Advancing Predictive Models of the Microbiome »
Elliott Gordon-Rodriguez · Thomas Quinn · John Cunningham -
2022 Poster: SIXO: Smoothing Inference with Twisted Objectives »
Dieterich Lawson · Allan Raventós · andrew warrington · Scott Linderman -
2022 Poster: Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs »
Julia Costacurta · Lea Duncker · Blue Sheffer · Winthrop Gillis · Caleb Weinreb · Jeffrey Markowitz · Sandeep R Datta · Alex Williams · Scott Linderman -
2022 Poster: Posterior and Computational Uncertainty in Gaussian Processes »
Jonathan Wenger · Geoff Pleiss · Marvin Pförtner · Philipp Hennig · John Cunningham -
2022 Poster: Deep Ensembles Work, But Are They Necessary? »
Taiga Abe · Estefany Kelly Buchanan · Geoff Pleiss · Richard Zemel · John Cunningham -
2021 Poster: Three-dimensional spike localization and improved motion correction for Neuropixels recordings »
Julien Boussard · Erdem Varol · Hyun Dong Lee · Nishchal Dethe · Liam Paninski -
2021 Poster: The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective »
Geoff Pleiss · John Cunningham -
2021 Poster: Posterior Collapse and Latent Variable Non-identifiability »
Yixin Wang · David Blei · John Cunningham -
2021 Poster: Rectangular Flows for Manifold Learning »
Anthony Caterini · Gabriel Loaiza-Ganem · Geoff Pleiss · John Cunningham -
2020 : Oral 05: Inverse articulated-body dynamics from video via variational sequential Monte-Carlo »
Dan Biderman -
2020 Poster: Point process models for sequence detection in high-dimensional neural spike trains »
Alex Williams · Anthony Degleris · Yixin Wang · Scott Linderman -
2020 Oral: Point process models for sequence detection in high-dimensional neural spike trains »
Alex Williams · Anthony Degleris · Yixin Wang · Scott Linderman -
2020 Poster: Deep Graph Pose: a semi-supervised deep graphical model for improved animal pose tracking »
Anqi Wu · Estefany Kelly Buchanan · Matthew Whiteway · Michael Schartner · Guido Meijer · Jean-Paul Noel · Erica Rodriguez · Claire Everett · Amy Norovich · Evan Schaffer · Neeli Mishra · C. Daniel Salzman · Dora Angelaki · Andrés Bendesky · The International Brain Laboratory The International Brain Laboratory · John Cunningham · Liam Paninski -
2020 Poster: Recurrent Switching Dynamical Systems Models for Multiple Interacting Neural Populations »
Joshua Glaser · Matthew Whiteway · John Cunningham · Liam Paninski · Scott Linderman -
2020 Poster: Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax »
Andres Potapczynski · Gabriel Loaiza-Ganem · John Cunningham -
2019 : Poster Session »
Pravish Sainath · Mohamed Akrout · Charles Delahunt · Nathan Kutz · Guangyu Robert Yang · Joseph Marino · L F Abbott · Nicolas Vecoven · Damien Ernst · andrew warrington · Michael Kagan · Kyunghyun Cho · Kameron Harris · Leopold Grinberg · John J. Hopfield · Dmitry Krotov · Taliah Muhammad · Erick Cobos · Edgar Walker · Jacob Reimer · Andreas Tolias · Alexander Ecker · Janaki Sheth · Yu Zhang · Maciej Wołczyk · Jacek Tabor · Szymon Maszke · Roman Pogodin · Dane Corneil · Wulfram Gerstner · Baihan Lin · Guillermo Cecchi · Jenna M Reinen · Irina Rish · Guillaume Bellec · Darjan Salaj · Anand Subramoney · Wolfgang Maass · Yueqi Wang · Ari Pakman · Jin Hyung Lee · Liam Paninski · Bryan Tripp · Colin Graber · Alex Schwing · Luke Prince · Gabriel Ocker · Michael Buice · Benjamin Lansdell · Konrad Kording · Jack Lindsey · Terrence Sejnowski · Matthew Farrell · Eric Shea-Brown · Nicolas Farrugia · Victor Nepveu · Jiwoong Im · Kristin Branson · Brian Hu · Ramakrishnan Iyer · Stefan Mihalas · Sneha Aenugu · Hananel Hazan · Sihui Dai · Tan Nguyen · Doris Tsao · Richard Baraniuk · Anima Anandkumar · Hidenori Tanaka · Aran Nayebi · Stephen Baccus · Surya Ganguli · Dean Pospisil · Eilif Muller · Jeffrey S Cheng · Gaël Varoquaux · Kamalaker Dadi · Dimitrios C Gklezakos · Rajesh PN Rao · Anand Louis · Christos Papadimitriou · Santosh Vempala · Naganand Yadati · Daniel Zdeblick · Daniela M Witten · Nicholas Roberts · Vinay Prabhu · Pierre Bellec · Poornima Ramesh · Jakob H Macke · Santiago Cadena · Guillaume Bellec · Franz Scherr · Owen Marschall · Robert Kim · Hannes Rapp · Marcio Fonseca · Oliver Armitage · Jiwoong Im · Thomas Hardcastle · Abhishek Sharma · Wyeth Bair · Adrian Valente · Shane Shang · Merav Stern · Rutuja Patil · Peter Wang · Sruthi Gorantla · Peter Stratton · Tristan Edwards · Jialin Lu · Martin Ester · Yurii Vlasov · Siavash Golkar -
2019 Workshop: Learning Meaningful Representations of Life »
Elizabeth Wood · Yakir Reshef · Jonathan Bloom · Jasper Snoek · Barbara Engelhardt · Scott Linderman · Suchi Saria · Alexander Wiltschko · Casey Greene · Chang Liu · Kresten Lindorff-Larsen · Debora Marks -
2019 Poster: Paraphrase Generation with Latent Bag of Words »
Yao Fu · Yansong Feng · John Cunningham -
2019 Poster: Deep Random Splines for Point Process Intensity Estimation of Neural Population Data »
Gabriel Loaiza-Ganem · Sean Perkins · Karen Schroeder · Mark Churchland · John Cunningham -
2019 Poster: Scalable Bayesian inference of dendritic voltage via spatiotemporal recurrent state space models »
Ruoxi Sun · Ian Kinsella · Scott Linderman · Liam Paninski -
2019 Poster: The continuous Bernoulli: fixing a pervasive error in variational autoencoders »
Gabriel Loaiza-Ganem · John Cunningham -
2019 Poster: Efficient characterization of electrically evoked responses for neural interfaces »
Nishal Shah · Sasidhar Madugula · Pawel Hottowy · Alexander Sher · Alan Litke · Liam Paninski · E.J. Chichilnisky -
2019 Oral: Scalable Bayesian inference of dendritic voltage via spatiotemporal recurrent state space models »
Ruoxi Sun · Ian Kinsella · Scott Linderman · Liam Paninski -
2019 Poster: Mutually Regressive Point Processes »
Ifigeneia Apostolopoulou · Scott Linderman · Kyle Miller · Artur Dubrawski -
2017 : Poster Session 1 »
Magdalena Fuchs · David Lung · Mathias Lechner · Kezhi Li · Andrew Gordus · Vivek Venkatachalam · Shivesh Chaudhary · Jan Hůla · David Rolnick · Scott Linderman · Gonzalo Mena · Liam Paninski · Netta Cohen -
2017 Spotlight: Deep Networks for Decoding Natural Images from Retinal Signals »
Nikhil Parthasarathy · Eleanor Batty · William Falcon · Thomas Rutten · Mohit Rajpal · E.J. Chichilnisky · Liam Paninski -
2017 Poster: Neural Networks for Efficient Bayesian Decoding of Natural Images from Retinal Neurons »
Nikhil Parthasarathy · Eleanor Batty · William Falcon · Thomas Rutten · Mohit Rajpal · E.J. Chichilnisky · Liam Paninski -
2017 Poster: OnACID: Online Analysis of Calcium Imaging Data in Real Time »
Andrea Giovannucci · Johannes Friedrich · Matt Kaufman · Anne Churchland · Dmitri Chklovskii · Liam Paninski · Eftychios Pnevmatikakis -
2017 Poster: YASS: Yet Another Spike Sorter »
Jin Hyung Lee · David Carlson · Hooshmand Shokri Razaghi · Weichi Yao · Georges A Goetz · Espen Hagen · Eleanor Batty · E.J. Chichilnisky · Gaute T. Einevoll · Liam Paninski -
2016 Poster: Linear dynamical neural population models through nonlinear embeddings »
Yuanjun Gao · Evan Archer · Liam Paninski · John Cunningham -
2016 Poster: Fast Active Set Methods for Online Spike Inference from Calcium Imaging »
Johannes Friedrich · Liam Paninski -
2016 Poster: Composing graphical models with neural networks for structured representations and fast inference »
Matthew Johnson · David Duvenaud · Alex Wiltschko · Ryan Adams · Sandeep R Datta -
2016 Poster: Automated scalable segmentation of neurons from multispectral images »
Uygar Sümbül · Douglas Roossien · Dawen Cai · Fei Chen · Nicholas Barry · John Cunningham · Edward Boyden · Liam Paninski -
2015 Poster: Bayesian Active Model Selection with an Application to Automated Audiometry »
Jacob Gardner · Gustavo Malkomes · Roman Garnett · Kilian Weinberger · Dennis Barbour · John Cunningham -
2015 Poster: High-dimensional neural spike train analysis with generalized count linear dynamical systems »
Yuanjun Gao · Lars Busing · Krishna V Shenoy · John Cunningham -
2015 Spotlight: High-dimensional neural spike train analysis with generalized count linear dynamical systems »
Yuanjun Gao · Lars Busing · Krishna V Shenoy · John Cunningham -
2014 Poster: Clustered factor analysis of multineuronal spike data »
Lars Buesing · Timothy A Machado · John P Cunningham · Liam Paninski -
2014 Spotlight: Clustered factor analysis of multineuronal spike data »
Lars Buesing · Timothy A Machado · John P Cunningham · Liam Paninski -
2013 Poster: A multi-agent control framework for co-adaptation in brain-computer interfaces »
Josh S Merel · Roy Fox · Tony Jebara · Liam Paninski -
2013 Poster: Bayesian Inference and Online Experimental Design for Mapping Neural Microcircuits »
Ben Shababo · Brooks Paige · Ari Pakman · Liam Paninski -
2013 Spotlight: Bayesian Inference and Online Experimental Design for Mapping Neural Microcircuits »
Ben Shababo · Brooks Paige · Ari Pakman · Liam Paninski -
2013 Poster: Auxiliary-variable Exact Hamiltonian Monte Carlo Samplers for Binary Distributions »
Ari Pakman · Liam Paninski -
2013 Poster: Sparse nonnegative deconvolution for compressive calcium imaging: algorithms and phase transitions »
Eftychios Pnevmatikakis · Liam Paninski -
2013 Poster: Robust learning of low-dimensional dynamics from large neural ensembles »
David Pfau · Eftychios Pnevmatikakis · Liam Paninski -
2011 Poster: Information Rates and Optimal Decoding in Large Neural Populations »
Kamiar Rahnama Rad · Liam Paninski -
2011 Spotlight: Information Rates and Optimal Decoding in Large Neural Populations »
Kamiar Rahnama Rad · Liam Paninski