Timezone: »

 
A robust estimator of mutual information for deep learning interpretability
Davide Piras · Hiranya Peiris · Andrew Pontzen · Luisa Lucie-Smith · Brian Nord · Ningyuan (Lillian) Guo

@
in Machine Learning and the Physical Sciences »

We develop the use of mutual information (MI), a well-established metric in information theory, to interpret the inner workings of deep learning models. To accurately estimate MI from a finite number of samples, we present GMM-MI, an algorithm based on Gaussian mixture models that can be applied to both discrete and continuous settings. GMM-MI is computationally efficient, robust to hyperparameter choices and provides the uncertainty on the MI estimate due to the finite sample size. We demonstrate the use of our MI estimator in the context of representation learning, working with synthetic data and physical datasets describing highly non-linear processes. We use GMM-MI to quantify both the level of disentanglement between the latent variables, and their association with relevant physical quantities, thus unlocking the interpretability of the latent representation.

[ Poster] 

Author Information

Davide Piras (University of Geneva)
Davide Piras

Postdoc in Geneva, working on machine learning applied to cosmology. Formerly at University College London.

Hiranya Peiris (University College London/Stockholm University)
Andrew Pontzen (University College London)
Luisa Lucie-Smith (Max Planck Institute for Astrophysics)
Brian Nord (Fermi National Accelerator Laboratory)
Ningyuan (Lillian) Guo (University College London)

More from the Same Authors