Timezone: »
Modern machine learning has seen the development of models of increasing complexity for high-dimensional real-world data, such as documents and images. Some of these models are implicit, meaning they generate samples without specifying a probability distribution function (e.g. GANs), and some are explicit, specifying a distribution function – with a potentially quite complex structure which may not admit efficient sampling or normalization. This tutorial will provide modern nonparametric tools for evaluating and benchmarking both implicit and explicit models. For implicit models, samples from the model are compared with real-world samples; for explicit models, a Stein operator is defined to compare the model to data samples without requiring a normalized probability distribution. In both cases, we also consider relative tests to choose the best of several incorrect models. We will emphasize interpretable tests throughout, where the way in which the model differs from the data is conveyed to the user.
Author Information
Wittawat Jitkrittum (Max Planck Institute for Intelligent Systems)
Danica J. Sutherland (TTIC)
D.J. Sutherland is a Research Assistant Professor at TTI-Chicago, and will begin as an Assistant Professor in UBC Computer Science in 2021. D.J. received a PhD from Carnegie Mellon in 2016, and was a postdoc at the Gatsby Unit, UCL, from 2016-19. D.J.’s research is largely in the connection between kernel methods and deep learning, with a focus on interfacing theory with methodology in areas including two-sample testing, generative models, distribution regression, and representation learning.
Arthur Gretton (Gatsby Unit, UCL)
Arthur Gretton is a Professor with the Gatsby Computational Neuroscience Unit at UCL. He received degrees in Physics and Systems Engineering from the Australian National University, and a PhD with Microsoft Research and the Signal Processing and Communications Laboratory at the University of Cambridge. He previously worked at the MPI for Biological Cybernetics, and at the Machine Learning Department, Carnegie Mellon University. Arthur's recent research interests in machine learning include the design and training of generative models, both implicit (e.g. GANs) and explicit (high/infinite dimensional exponential family models), nonparametric hypothesis testing, and kernel methods. He has been an associate editor at IEEE Transactions on Pattern Analysis and Machine Intelligence from 2009 to 2013, an Action Editor for JMLR since April 2013, an Area Chair for NeurIPS in 2008 and 2009, a Senior Area Chair for NeurIPS in 2018, an Area Chair for ICML in 2011 and 2012, and a member of the COLT Program Committee in 2013. Arthur was program chair for AISTATS in 2016 (with Christian Robert), tutorials chair for ICML 2018 (with Ruslan Salakhutdinov), workshops chair for ICML 2019 (with Honglak Lee), program chair for the Dali workshop in 2019 (with Krikamol Muandet and Shakir Mohammed), and co-organsier of the Machine Learning Summer School 2019 in London (with Marc Deisenroth).
More from the Same Authors
-
2021 : Kernel Methods for Multistage Causal Inference: Mediation Analysis and Dynamic Treatment Effects »
Rahul Singh · Ritsugen Jo · Arthur Gretton -
2021 : Composite Goodness-of-fit Tests with Kernels »
Oscar Key · Tamara Fernandez · Arthur Gretton · Francois-Xavier Briol -
2022 Poster: Optimal Rates for Regularized Conditional Mean Embedding Learning »
Zhu Li · Dimitri Meunier · Mattes Mollenhauer · Arthur Gretton -
2022 Poster: KSD Aggregated Goodness-of-fit Test »
Antonin Schrab · Benjamin Guedj · Arthur Gretton -
2022 Poster: Efficient Aggregated Kernel Tests using Incomplete $U$-statistics »
Antonin Schrab · Ilmun Kim · Benjamin Guedj · Arthur Gretton -
2022 Poster: Post-hoc estimators for learning to defer to an expert »
Harikrishna Narasimhan · Wittawat Jitkrittum · Aditya Menon · Ankit Rawat · Sanjiv Kumar -
2021 Workshop: Machine Learning Meets Econometrics (MLECON) »
David Bruns-Smith · Arthur Gretton · Limor Gultchin · Niki Kilbertus · Krikamol Muandet · Evan Munro · Angela Zhou -
2021 Poster: KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support »
Pierre Glaser · Michael Arbel · Arthur Gretton -
2021 Poster: Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluation »
Ritsugen Jo · Heishiro Kanagawa · Arthur Gretton -
2021 Poster: Self-Supervised Learning with Kernel Dependence Maximization »
Yazhe Li · Roman Pogodin · Danica J. Sutherland · Arthur Gretton -
2020 Poster: A Non-Asymptotic Analysis for Stein Variational Gradient Descent »
Anna Korba · Adil Salim · Michael Arbel · Giulia Luise · Arthur Gretton -
2020 Poster: Learning Kernel Tests Without Data Splitting »
Jonas Kübler · Wittawat Jitkrittum · Bernhard Schölkopf · Krikamol Muandet -
2020 Poster: A kernel test for quasi-independence »
Tamara Fernandez · Wenkai Xu · Marc Ditzhaus · Arthur Gretton -
2020 Spotlight: A kernel test for quasi-independence »
Tamara Fernandez · Wenkai Xu · Marc Ditzhaus · Arthur Gretton -
2019 Poster: Exponential Family Estimation via Adversarial Dynamics Embedding »
Bo Dai · Zhen Liu · Hanjun Dai · Niao He · Arthur Gretton · Le Song · Dale Schuurmans -
2019 Poster: Maximum Mean Discrepancy Gradient Flow »
Michael Arbel · Anna Korba · Adil Salim · Arthur Gretton -
2019 Poster: Kernel Instrumental Variable Regression »
Rahul Singh · Maneesh Sahani · Arthur Gretton -
2019 Poster: Kernel Stein Tests for Multiple Model Comparison »
Jen Ning Lim · Makoto Yamada · Bernhard Schölkopf · Wittawat Jitkrittum -
2019 Poster: Fisher Efficient Inference of Intractable Models »
Song Liu · Takafumi Kanamori · Wittawat Jitkrittum · Yu Chen -
2019 Oral: Kernel Instrumental Variable Regression »
Rahul Singh · Maneesh Sahani · Arthur Gretton -
2018 Poster: Informative Features for Model Comparison »
Wittawat Jitkrittum · Heishiro Kanagawa · Patsorn Sangkloy · James Hays · Bernhard Schölkopf · Arthur Gretton -
2018 Poster: BRUNO: A Deep Recurrent Model for Exchangeable Data »
Iryna Korshunova · Jonas Degrave · Ferenc Huszar · Yarin Gal · Arthur Gretton · Joni Dambre -
2018 Poster: On gradient regularizers for MMD GANs »
Michael Arbel · Danica J. Sutherland · Mikołaj Bińkowski · Arthur Gretton -
2017 : A Linear-Time Kernel Goodness-of-Fit Test (NIPS best paper) »
Wittawat Jitkrittum -
2017 : Conditional Densities and Efficient Models in Infinite Exponential Families »
Arthur Gretton -
2017 Poster: A Linear-Time Kernel Goodness-of-Fit Test »
Wittawat Jitkrittum · Wenkai Xu · Zoltan Szabo · Kenji Fukumizu · Arthur Gretton -
2017 Oral: A Linear-Time Kernel Goodness-of-Fit Test »
Wittawat Jitkrittum · Wenkai Xu · Zoltan Szabo · Kenji Fukumizu · Arthur Gretton -
2016 Workshop: Adaptive and Scalable Nonparametric Methods in Machine Learning »
Aaditya Ramdas · Arthur Gretton · Bharath Sriperumbudur · Han Liu · John Lafferty · Samory Kpotufe · Zoltán Szabó -
2016 : Discussion panel »
Ian Goodfellow · Soumith Chintala · Arthur Gretton · Sebastian Nowozin · Aaron Courville · Yann LeCun · Emily Denton -
2016 : Learning features to distinguish distributions »
Arthur Gretton -
2016 Oral: Interpretable Distribution Features with Maximum Testing Power »
Wittawat Jitkrittum · Zoltán Szabó · Kacper P Chwialkowski · Arthur Gretton -
2016 Poster: Interpretable Distribution Features with Maximum Testing Power »
Wittawat Jitkrittum · Zoltán Szabó · Kacper P Chwialkowski · Arthur Gretton -
2015 : *Arthur Gretton* Learning with Probabilities as Inputs, Using Kernels »
Arthur Gretton -
2015 Poster: Bayesian Manifold Learning: The Locally Linear Latent Variable Model (LL-LVM) »
Mijung Park · Wittawat Jitkrittum · Ahmad Qamar · Zoltan Szabo · Lars Buesing · Maneesh Sahani -
2015 Poster: Gradient-free Hamiltonian Monte Carlo with Efficient Kernel Exponential Families »
Heiko Strathmann · Dino Sejdinovic · Samuel Livingstone · Zoltan Szabo · Arthur Gretton -
2015 Poster: Fast Two-Sample Testing with Analytic Representations of Probability Measures »
Kacper P Chwialkowski · Aaditya Ramdas · Dino Sejdinovic · Arthur Gretton -
2014 Workshop: Modern Nonparametrics 3: Automating the Learning Pipeline »
Eric Xing · Mladen Kolar · Arthur Gretton · Samory Kpotufe · Han Liu · Zoltán Szabó · Alan Yuille · Andrew G Wilson · Ryan Tibshirani · Sasha Rakhlin · Damian Kozbur · Bharath Sriperumbudur · David Lopez-Paz · Kirthevasan Kandasamy · Francesco Orabona · Andreas Damianou · Wacha Bounliphone · Yanshuai Cao · Arijit Das · Yingzhen Yang · Giulia DeSalvo · Dmitry Storcheus · Roberto Valerio -
2014 Poster: A Wild Bootstrap for Degenerate Kernel Tests »
Kacper P Chwialkowski · Dino Sejdinovic · Arthur Gretton -
2014 Oral: A Wild Bootstrap for Degenerate Kernel Tests »
Kacper P Chwialkowski · Dino Sejdinovic · Arthur Gretton -
2013 Workshop: New Directions in Transfer and Multi-Task: Learning Across Domains and Tasks »
Urun Dogan · Marius Kloft · Tatiana Tommasi · Francesco Orabona · Massimiliano Pontil · Sinno Jialin Pan · Shai Ben-David · Arthur Gretton · Fei Sha · Marco Signoretto · Rajhans Samdani · Yun-Qian Miao · Mohammad Gheshlaghi azar · Ruth Urner · Christoph Lampert · Jonathan How -
2013 Workshop: Modern Nonparametric Methods in Machine Learning »
Arthur Gretton · Mladen Kolar · Samory Kpotufe · John Lafferty · Han Liu · Bernhard Schölkopf · Alexander Smola · Rob Nowak · Mikhail Belkin · Lorenzo Rosasco · peter bickel · Yue Zhao -
2013 Poster: B-test: A Non-parametric, Low Variance Kernel Two-sample Test »
Wojciech Zaremba · Arthur Gretton · Matthew B Blaschko -
2013 Poster: A Kernel Test for Three-Variable Interactions »
Dino Sejdinovic · Arthur Gretton · Wicher Bergsma -
2013 Oral: A Kernel Test for Three-Variable Interactions »
Dino Sejdinovic · Arthur Gretton · Wicher Bergsma -
2012 Workshop: Confluence between Kernel Methods and Graphical Models »
Le Song · Arthur Gretton · Alexander Smola -
2012 Workshop: Modern Nonparametric Methods in Machine Learning »
Sivaraman Balakrishnan · Arthur Gretton · Mladen Kolar · John Lafferty · Han Liu · Tong Zhang -
2012 Poster: Optimal kernel choice for large-scale two-sample tests »
Arthur Gretton · Bharath Sriperumbudur · Dino Sejdinovic · Heiko Strathmann · Sivaraman Balakrishnan · Massimiliano Pontil · Kenji Fukumizu -
2011 Poster: Kernel Bayes' Rule »
Kenji Fukumizu · Le Song · Arthur Gretton -
2010 Workshop: Low-rank Methods for Large-scale Machine Learning »
Arthur Gretton · Michael W Mahoney · Mehryar Mohri · Ameet S Talwalkar -
2009 Workshop: Temporal Segmentation: Perspectives from Statistics, Machine Learning, and Signal Processing »
Stephane Canu · Olivier Cappé · Arthur Gretton · Zaid Harchaoui · Alain Rakotomamonjy · Jean-Philippe Vert -
2009 Workshop: Large-Scale Machine Learning: Parallelism and Massive Datasets »
Alexander Gray · Arthur Gretton · Alexander Smola · Joseph E Gonzalez · Carlos Guestrin -
2009 Session: Oral session 10: Neural Modeling and Imaging »
Arthur Gretton -
2009 Poster: Kernel Choice and Classifiability for RKHS Embeddings of Probability Distributions »
Bharath Sriperumbudur · Kenji Fukumizu · Arthur Gretton · Gert Lanckriet · Bernhard Schölkopf -
2009 Oral: Kernel Choice and Classifiability for RKHS Embeddings of Probability Distributions »
Bharath Sriperumbudur · Kenji Fukumizu · Arthur Gretton · Gert Lanckriet · Bernhard Schölkopf -
2009 Poster: Nonlinear directed acyclic structure learning with weakly additive noise models »
Robert E Tillman · Arthur Gretton · Peter Spirtes -
2009 Poster: A Fast, Consistent Kernel Two-Sample Test »
Arthur Gretton · Kenji Fukumizu · Zaid Harchaoui · Bharath Sriperumbudur -
2009 Spotlight: A Fast, Consistent Kernel Two-Sample Test »
Arthur Gretton · Kenji Fukumizu · Zaid Harchaoui · Bharath Sriperumbudur -
2008 Workshop: Kernel Learning: Automatic Selection of Optimal Kernels »
Corinna Cortes · Arthur Gretton · Gert Lanckriet · Mehryar Mohri · Afshin Rostamizadeh -
2008 Poster: Kernel Measures of Independence for non-iid Data »
Xinhua Zhang · Le Song · Arthur Gretton · Alexander Smola -
2008 Poster: Characteristic Kernels on Groups and Semigroups »
Kenji Fukumizu · Bharath Sriperumbudur · Arthur Gretton · Bernhard Schölkopf -
2008 Spotlight: Kernel Measures of Independence for non-iid Data »
Xinhua Zhang · Le Song · Arthur Gretton · Alexander Smola -
2008 Oral: Characteristic Kernels on Groups and Semigroups »
Kenji Fukumizu · Bharath Sriperumbudur · Arthur Gretton · Bernhard Schölkopf -
2008 Session: Oral session 2: Sensorimotor Control »
Arthur Gretton -
2008 Poster: Learning Taxonomies by Dependence Maximization »
Matthew B Blaschko · Arthur Gretton -
2007 Workshop: Representations and Inference on Probability Distributions »
Kenji Fukumizu · Arthur Gretton · Alexander Smola -
2007 Spotlight: Kernel Measures of Conditional Dependence »
Kenji Fukumizu · Arthur Gretton · Xiaohai Sun · Bernhard Schölkopf -
2007 Poster: Kernel Measures of Conditional Dependence »
Kenji Fukumizu · Arthur Gretton · Xiaohai Sun · Bernhard Schölkopf -
2007 Spotlight: A Kernel Statistical Test of Independence »
Arthur Gretton · Kenji Fukumizu · Choon Hui Teo · Le Song · Bernhard Schölkopf · Alexander Smola -
2007 Oral: Colored Maximum Variance Unfolding »
Le Song · Alexander Smola · Karsten Borgwardt · Arthur Gretton -
2007 Poster: Colored Maximum Variance Unfolding »
Le Song · Alexander Smola · Karsten Borgwardt · Arthur Gretton -
2007 Poster: A Kernel Statistical Test of Independence »
Arthur Gretton · Kenji Fukumizu · Choon Hui Teo · Le Song · Bernhard Schölkopf · Alexander Smola -
2006 Poster: A Kernel Method for the Two-Sample-Problem »
Arthur Gretton · Karsten Borgwardt · Malte J Rasch · Bernhard Schölkopf · Alexander Smola -
2006 Poster: Correcting Sample Selection Bias by Unlabeled Data »
Jiayuan Huang · Alexander Smola · Arthur Gretton · Karsten Borgwardt · Bernhard Schölkopf -
2006 Spotlight: Correcting Sample Selection Bias by Unlabeled Data »
Jiayuan Huang · Alexander Smola · Arthur Gretton · Karsten Borgwardt · Bernhard Schölkopf -
2006 Talk: A Kernel Method for the Two-Sample-Problem »
Arthur Gretton · Karsten Borgwardt · Malte J Rasch · Bernhard Schölkopf · Alexander Smola