Timezone: »
Mutual information (MI) is a fundamental measure of statistical dependence, with a myriad of applications to information theory, statistics, and machine learning. While it possesses many desirable structural properties, the estimation of high-dimensional MI from samples suffers from the curse of dimensionality. Motivated by statistical scalability to high dimensions, this paper proposes sliced MI (SMI) as a surrogate measure of dependence. SMI is defined as an average of MI terms between one-dimensional random projections. We show that it preserves many of the structural properties of classic MI, while gaining scalable computation and efficient estimation from samples. Furthermore, and in contrast to classic MI, SMI can grow as a result of deterministic transformations. This enables leveraging SMI for feature extraction by optimizing it over processing functions of raw data to identify useful representations thereof. Our theory is supported by numerical studies of independence testing and feature extraction, which demonstrate the potential gains SMI offers over classic MI for high-dimensional inference.
Author Information
Ziv Goldfeld (Cornell University)
Kristjan Greenewald (MIT-IBM Watson AI Lab; IBM Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: Sliced Mutual Information: A Scalable Measure of Statistical Dependence »
Dates n/a. Room
More from the Same Authors
-
2021 Spotlight: Measuring Generalization with Optimal Transport »
Ching-Yao Chuang · Youssef Mroueh · Kristjan Greenewald · Antonio Torralba · Stefanie Jegelka -
2023 Poster: Outlier-Robust Wasserstein DRO »
Sloan Nietert · Ziv Goldfeld · Soroosh Shafieezadeh-Abadeh -
2023 Poster: Max-Sliced Mutual Information »
Dor Tsur · Ziv Goldfeld · Kristjan Greenewald -
2023 Workshop: Optimal Transport and Machine Learning »
Anna Korba · Aram-Alexandre Pooladian · Charlotte Bunne · David Alvarez-Melis · Marco Cuturi · Ziv Goldfeld -
2022 Poster: $k$-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension »
Ziv Goldfeld · Kristjan Greenewald · Theshani Nuradha · Galen Reeves -
2022 Poster: Statistical, Robustness, and Computational Guarantees for Sliced Wasserstein Distances »
Sloan Nietert · Ziv Goldfeld · Ritwik Sadhu · Kengo Kato -
2021 Poster: Measuring Generalization with Optimal Transport »
Ching-Yao Chuang · Youssef Mroueh · Kristjan Greenewald · Antonio Torralba · Stefanie Jegelka -
2020 Poster: Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance »
Ziv Goldfeld · Kristjan Greenewald · Kengo Kato