Timezone: »
Maximum entropy models have become popular statistical models in neuroscience and other areas in biology, and can be useful tools for obtaining estimates of mu- tual information in biological systems. However, maximum entropy models fit to small data sets can be subject to sampling bias; i.e. the true entropy of the data can be severely underestimated. Here we study the sampling properties of estimates of the entropy obtained from maximum entropy models. We show that if the data is generated by a distribution that lies in the model class, the bias is equal to the number of parameters divided by twice the number of observations. However, in practice, the true distribution is usually outside the model class, and we show here that this misspecification can lead to much larger bias. We provide a perturba- tive approximation of the maximally expected bias when the true model is out of model class, and we illustrate our results using numerical simulations of an Ising model; i.e. the second-order maximum entropy distribution on binary data.
Author Information
Jakob H Macke (University of Tübingen & MPI IS Tübingen)
Iain Murray (University of Edinburgh)
Iain Murray is a SICSA Lecturer in Machine Learning at the University of Edinburgh. Iain was introduced to machine learning by David MacKay and Zoubin Ghahramani, both previous NIPS tutorial speakers. He obtained his PhD in 2007 from the Gatsby Computational Neuroscience Unit at UCL. His thesis on Monte Carlo methods received an honourable mention for the ISBA Savage Award. He was a commonwealth fellow in Machine Learning at the University of Toronto, before moving to Edinburgh in 2010. Iain's research interests include building flexible probabilistic models of data, and probabilistic inference from indirect and uncertain observations. Iain is passionate about teaching. He has lectured at several Summer schools, is listed in the top 15 authors on videolectures.net, and was awarded the EUSA Van Heyningen Award for Teaching in Science and Engineering in 2015.
Peter E Latham (Gatsby Unit, UCL)
More from the Same Authors
-
2021 Spotlight: Maximum Likelihood Training of Score-Based Diffusion Models »
Yang Song · Conor Durkan · Iain Murray · Stefano Ermon -
2023 Poster: Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation »
Richard Gao · Michael Deistler · Jakob H Macke -
2023 Poster: Meta-learning families of plasticity rules in recurrent spiking networks using simulation-based inference »
Basile Confavreux · Poornima Ramesh · Pedro Goncalves · Jakob H Macke · Tim Vogels -
2023 Poster: Flow Matching for Scalable Simulation-Based Inference »
Jonas Wildberger · Maximilian Dax · Simon Buchholz · Stephen Green · Jakob H Macke · Bernhard Schölkopf -
2022 Poster: On the Stability and Scalability of Node Perturbation Learning »
Naoki Hiratani · Yash Mehta · Timothy Lillicrap · Peter E Latham -
2022 Poster: Truncated proposals for scalable and hassle-free simulation-based inference »
Michael Deistler · Pedro Goncalves · Jakob H Macke -
2022 Poster: Efficient identification of informative features in simulation-based inference »
Jonas Beck · Michael Deistler · Yves Bernaerts · Jakob H Macke · Philipp Berens -
2021 Poster: Powerpropagation: A sparsity inducing weight reparameterisation »
Jonathan Richard Schwarz · Siddhant Jayakumar · Razvan Pascanu · Peter E Latham · Yee Teh -
2021 Poster: Maximum Likelihood Training of Score-Based Diffusion Models »
Yang Song · Conor Durkan · Iain Murray · Stefano Ermon -
2021 Poster: Towards Biologically Plausible Convolutional Networks »
Roman Pogodin · Yash Mehta · Timothy Lillicrap · Peter E Latham -
2020 Poster: Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks »
Roman Pogodin · Peter E Latham -
2019 : Poster Session »
Pravish Sainath · Mohamed Akrout · Charles Delahunt · Nathan Kutz · Guangyu Robert Yang · Joseph Marino · L F Abbott · Nicolas Vecoven · Damien Ernst · andrew warrington · Michael Kagan · Kyunghyun Cho · Kameron Harris · Leopold Grinberg · John J. Hopfield · Dmitry Krotov · Taliah Muhammad · Erick Cobos · Edgar Walker · Jacob Reimer · Andreas Tolias · Alexander Ecker · Janaki Sheth · Yu Zhang · Maciej Wołczyk · Jacek Tabor · Szymon Maszke · Roman Pogodin · Dane Corneil · Wulfram Gerstner · Baihan Lin · Guillermo Cecchi · Jenna M Reinen · Irina Rish · Guillaume Bellec · Darjan Salaj · Anand Subramoney · Wolfgang Maass · Yueqi Wang · Ari Pakman · Jin Hyung Lee · Liam Paninski · Bryan Tripp · Colin Graber · Alex Schwing · Luke Prince · Gabriel Ocker · Michael Buice · Benjamin Lansdell · Konrad Kording · Jack Lindsey · Terrence Sejnowski · Matthew Farrell · Eric Shea-Brown · Nicolas Farrugia · Victor Nepveu · Jiwoong Im · Kristin Branson · Brian Hu · Ramakrishnan Iyer · Stefan Mihalas · Sneha Aenugu · Hananel Hazan · Sihui Dai · Tan Nguyen · Doris Tsao · Richard Baraniuk · Anima Anandkumar · Hidenori Tanaka · Aran Nayebi · Stephen Baccus · Surya Ganguli · Dean Pospisil · Eilif Muller · Jeffrey S Cheng · Gaël Varoquaux · Kamalaker Dadi · Dimitrios C Gklezakos · Rajesh PN Rao · Anand Louis · Christos Papadimitriou · Santosh Vempala · Naganand Yadati · Daniel Zdeblick · Daniela M Witten · Nicholas Roberts · Vinay Prabhu · Pierre Bellec · Poornima Ramesh · Jakob H Macke · Santiago Cadena · Guillaume Bellec · Franz Scherr · Owen Marschall · Robert Kim · Hannes Rapp · Marcio Fonseca · Oliver Armitage · Jiwoong Im · Thomas Hardcastle · Abhishek Sharma · Wyeth Bair · Adrian Valente · Shane Shang · Merav Stern · Rutuja Patil · Peter Wang · Sruthi Gorantla · Peter Stratton · Tristan Edwards · Jialin Lu · Martin Ester · Yurii Vlasov · Siavash Golkar -
2019 Poster: Neural Spline Flows »
Conor Durkan · Artur Bekasov · Iain Murray · George Papamakarios -
2019 Poster: Intrinsic dimension of data representations in deep neural networks »
Alessio Ansuini · Alessandro Laio · Jakob H Macke · Davide Zoccolan -
2017 : Panel session »
Iain Murray · Max Welling · Juan Carrasquilla · Anatole von Lilienfeld · Gilles Louppe · Kyle Cranmer -
2017 : Invited talk 3: Learning priors, likelihoods, or posteriors »
Iain Murray -
2017 : Invited talk: Iain Murray (TBA) »
Iain Murray -
2017 Spotlight: Fast amortized inference of neural activity from calcium imaging data with variational autoencoders »
Artur Speiser · Jinyao Yan · Evan Archer · Lars Buesing · Srinivas C Turaga · Jakob H Macke -
2017 Oral: Masked Autoregressive Flow for Density Estimation »
George Papamakarios · Iain Murray · Theo Pavlakou -
2017 Poster: Masked Autoregressive Flow for Density Estimation »
George Papamakarios · Iain Murray · Theo Pavlakou -
2017 Poster: Fast amortized inference of neural activity from calcium imaging data with variational autoencoders »
Artur Speiser · Jinyao Yan · Evan Archer · Lars Buesing · Srinivas C Turaga · Jakob H Macke -
2017 Poster: Extracting low-dimensional dynamics from multiple large-scale neural population recordings by learning to predict correlations »
Marcel Nonnenmacher · Srinivas C Turaga · Jakob H Macke -
2017 Poster: Flexible statistical inference for mechanistic models of neural dynamics »
Jan-Matthis Lueckmann · Pedro Goncalves · Giacomo Bassetto · Kaan Öcal · Marcel Nonnenmacher · Jakob H Macke -
2016 : From Brains to Bits and Back Again »
Yoshua Bengio · Terrence Sejnowski · Christos H Papadimitriou · Jakob H Macke · Demis Hassabis · Alyson Fletcher · Andreas Tolias · Jascha Sohl-Dickstein · Konrad P Koerding -
2016 Workshop: Brains and Bits: Neuroscience meets Machine Learning »
Alyson Fletcher · Eva Dyer · Jascha Sohl-Dickstein · Joshua T Vogelstein · Konrad Koerding · Jakob H Macke -
2016 Poster: Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation »
George Papamakarios · Iain Murray -
2015 : Correlations and Signatures of Criticality in Neural Population Models »
Jakob H Macke -
2015 Workshop: Statistical Methods for Understanding Neural Systems »
Alyson Fletcher · Jakob H Macke · Ryan Adams · Jascha Sohl-Dickstein -
2015 Poster: Unlocking neural population non-stationarities using hierarchical dynamics models »
Mijung Park · Gergo Bohner · Jakob H Macke -
2015 Tutorial: Monte Carlo Inference Methods »
Iain Murray -
2014 Workshop: Large scale optical physiology: From data-acquisition to models of neural coding »
Il Memming Park · Jakob H Macke · Ferran Diego Andilla · Eftychios Pnevmatikakis · Jeremy Freeman -
2014 Poster: A Bayesian model for identifying hierarchically organised states in neural population activity »
Patrick Putzky · Florian Franzen · Giacomo Bassetto · Jakob H Macke -
2014 Spotlight: A Bayesian model for identifying hierarchically organised states in neural population activity »
Patrick Putzky · Florian Franzen · Giacomo Bassetto · Jakob H Macke -
2014 Poster: Low-dimensional models of neural population activity in sensory cortical circuits »
Evan Archer · Urs Koster · Jonathan W Pillow · Jakob H Macke -
2013 Workshop: Acquiring and Analyzing the Activity of Large Neural Ensembles »
Srinivas C Turaga · Lars Buesing · Maneesh Sahani · Jakob H Macke -
2013 Poster: Inferring neural population dynamics from multiple partial recordings of the same neural circuit »
Srinivas C Turaga · Lars Buesing · Adam M Packer · Henry Dalgleish · Noah Pettit · Michael Hausser · Jakob H Macke -
2013 Poster: Demixing odors - fast inference in olfaction »
Agnieszka Grabska-Barwinska · Jeff Beck · Alexandre Pouget · Peter E Latham -
2013 Poster: RNADE: The real-valued neural autoregressive density-estimator »
Benigno Uria · Iain Murray · Hugo Larochelle -
2013 Spotlight: Demixing odors - fast inference in olfaction »
Agnieszka Grabska-Barwinska · Jeff Beck · Alexandre Pouget · Peter E Latham -
2013 Spotlight: Inferring neural population dynamics from multiple partial recordings of the same neural circuit »
Srinivas C Turaga · Lars Buesing · Adam M Packer · Henry Dalgleish · Noah Pettit · Michael Hausser · Jakob H Macke -
2012 Poster: Spectral learning of linear dynamics from generalised-linear observations with application to neural population data »
Lars Buesing · Jakob H Macke · Maneesh Sahani -
2012 Oral: Spectral learning of linear dynamics from generalised-linear observations with application to neural population data »
Lars Buesing · Jakob H Macke · Maneesh Sahani -
2011 Oral: Empirical models of spiking in neural populations »
Jakob H Macke · Lars Buesing · John P Cunningham · Byron M Yu · Krishna V Shenoy · Maneesh Sahani -
2011 Poster: Empirical models of spiking in neural populations »
Jakob H Macke · Lars Buesing · John P Cunningham · Byron M Yu · Krishna V Shenoy · Maneesh Sahani -
2010 Workshop: Monte Carlo Methods for Bayesian Inference in Modern Day Applications »
Ryan Adams · Mark A Girolami · Iain Murray -
2010 Oral: Slice sampling covariance hyperparameters of latent Gaussian models »
Iain Murray · Ryan Adams -
2010 Poster: Slice sampling covariance hyperparameters of latent Gaussian models »
Iain Murray · Ryan Adams -
2010 Session: Spotlights Session 5 »
Iain Murray -
2010 Session: Oral Session 5 »
Iain Murray -
2009 Poster: Bayesian estimation of orientation preference maps »
Jakob H Macke · Sebastian Gerwinn · Leonard White · Matthias Kaschube · Matthias Bethge -
2008 Poster: Comparing model predictions of response bias and variance in cue combination »
Rama Natarajan · Iain Murray · Ladan Shams · Richard Zemel -
2008 Poster: The Gaussian Process Density Sampler »
Ryan Adams · Iain Murray · David MacKay -
2008 Spotlight: The Gaussian Process Density Sampler »
Ryan Adams · Iain Murray · David MacKay -
2008 Poster: Evaluating probabilities under high-dimensional latent variable models »
Iain Murray · Russ Salakhutdinov -
2008 Spotlight: Evaluating probabilities under high-dimensional latent variable models »
Iain Murray · Russ Salakhutdinov -
2007 Oral: Neural characterization in partially observed populations of spiking neurons »
Jonathan W Pillow · Peter E Latham -
2007 Oral: Bayesian Inference for Spiking Neuron Models with a Sparsity Prior »
Sebastian Gerwinn · Jakob H Macke · Matthias Seeger · Matthias Bethge -
2007 Poster: Bayesian Inference for Spiking Neuron Models with a Sparsity Prior »
Sebastian Gerwinn · Jakob H Macke · Matthias Seeger · Matthias Bethge -
2007 Poster: Neural characterization in partially observed populations of spiking neurons »
Jonathan W Pillow · Peter E Latham -
2007 Poster: Receptive Fields without Spike-Triggering »
Jakob H Macke · Günther Zeck · Matthias Bethge -
2006 Poster: Inducing Metric Violations in Human Similarity Judgements »
Julian Laub · Jakob H Macke · Klaus-Robert Müller · Felix A Wichmann