Timezone: »
Despite the recent popularity of neural network-based solvers for optimal transport (OT), there is no standard quantitative way to evaluate their performance. In this paper, we address this issue for quadratic-cost transport---specifically, computation of the Wasserstein-2 distance, a commonly-used formulation of optimal transport in machine learning. To overcome the challenge of computing ground truth transport maps between continuous measures needed to assess these solvers, we use input-convex neural networks (ICNN) to construct pairs of measures whose ground truth OT maps can be obtained analytically. This strategy yields pairs of continuous benchmark measures in high-dimensional spaces such as spaces of images. We thoroughly evaluate existing optimal transport solvers using these benchmark measures. Even though these solvers perform well in downstream tasks, many do not faithfully recover optimal transport maps. To investigate the cause of this discrepancy, we further test the solvers in a setting of image generation. Our study reveals crucial limitations of existing solvers and shows that increased OT accuracy does not necessarily correlate to better results downstream.
Author Information
Alexander Korotin (Skolkovo Institute of Science and Technology)
Lingxiao Li (MIT)
Aude Genevay (MIT)
Justin Solomon (MIT)
Alexander Filippov (Huawei Noah's Ark Lab)
Evgeny Burnaev (Skoltech)
Evgeny is an experienced scientist working at the interface between machine learning and applied engineering problems. He obtained his Master’s degree in Applied Physics and Mathematics from the Moscow Institute of Physics and Technology in 2006. After successfully defending his PhD thesis in Foundations of Computer Science at the Institute for Information Transmission Problem RAS (IITP RAS) in 2008, Evgeny stayed with the Institute as a head of IITP Data Analysis and Modeling group. Today, Evgeny’s research interests encompass the areas of regression based on Gaussian Processes, bootstrap, confidence sets and conformal predictors, volatility modeling and nonparametric estimation, statistical decisions and rapid detection of anomalies in complex multicomponent systems. Evgeny always demonstrated a deep fundamental knowledge and engineer-like thinking that enabled him to effectively use methods of statistics, machine learning and predictive modeling to deal with practical tasks in hi-tech industries, primarily aerospace, medicine and life sciences. He carried out a number of successful industrial projects with Airbus, Eurocopter and Sahara Force India Formula 1 team among others. The corresponding data analysis algorithms, developed by Evgeny and his group at IITP, formed a core of the algorithmic software library for surrogate modeling and optimization. Thanks to the developed functionality, engineers can construct fast mathematical approximations to long running computer codes (realizing physical models) based on available data and perform design space exploration for trade-off studies. The software library passed the final Technology Readiness Level 6 certification in Airbus. According to Airbus experts, application of the library “provides the reduction of up to 10% of lead time and cost in several areas of the aircraft design process”. Nowadays several dozens of Airbus departments use it. Later a spin-off company developed a Software platform for Design Space Exploration with GUI based on this algorithmic core. Evgeny has also a considerable teaching experience both in Russian and English. He has developed and taught various undergraduate and graduate courses in applied mathematics at MIPT, IITP, Yandex School of Data Analysis and the Humboldt University of Berlin, as well as mini courses on application of machine learning in engineering multidisciplinary modeling and optimization for technological companies such as Astrium, Safran, SAFT, CNES, etc. Before joining Skoltech, Evgeny was a Lecturer at Yandex School of Data Analysis, Associate Professor and Vice Chairman of Information Transmission Problems and Data Analysis Chair at MIPT, data analysis expert at DATADVANCE llc., and head of IITP Data Analysis and Predictive Modeling Lab. At Skoltech, Evgeny is actively engaged in the development of CDISE educational and research programs, and continues his research in the areas of development of theoretical tools for estimation of change-point algorithms’ performance, effective algorithms for anomaly detection and failures prediction, analysis of their properties, and development of a core library for anomaly detection and failures prediction.
More from the Same Authors
-
2022 Poster: Wasserstein Iterative Networks for Barycenter Estimation »
Alexander Korotin · Vage Egiazarian · Lingxiao Li · Evgeny Burnaev -
2022 Poster: Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport? »
Alexander Korotin · Alexander Kolesov · Evgeny Burnaev -
2022 Spotlight: Lightning Talks 6B-2 »
Alexander Korotin · Jinyuan Jia · Weijian Deng · Shi Feng · Maying Shen · Denizalp Goktas · Fang-Yi Yu · Alexander Kolesov · Sadie Zhao · Stephen Gould · Hongxu Yin · Wenjie Qu · Liang Zheng · Evgeny Burnaev · Amy Greenwald · Neil Gong · Pavlo Molchanov · Yiling Chen · Lei Mao · Jianna Liu · Jose M. Alvarez -
2022 Spotlight: Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport? »
Alexander Korotin · Alexander Kolesov · Evgeny Burnaev -
2022 Spotlight: Lightning Talks 2A-4 »
Sarthak Mittal · Richard Grumitt · Zuoyu Yan · Lihao Wang · Dongsheng Wang · Alexander Korotin · Jiangxin Sun · Ankit Gupta · Vage Egiazarian · Tengfei Ma · Yi Zhou · Yishi Xu · Albert Gu · Biwei Dai · Chunyu Wang · Yoshua Bengio · Uros Seljak · Miaoge Li · Guillaume Lajoie · Yiqun Wang · Liangcai Gao · Lingxiao Li · Jonathan Berant · Huang Hu · Xiaoqing Zheng · Zhibin Duan · Hanjiang Lai · Evgeny Burnaev · Zhi Tang · Zhi Jin · Xuanjing Huang · Chaojie Wang · Yusu Wang · Jian-Fang Hu · Bo Chen · Chao Chen · Hao Zhou · Mingyuan Zhou -
2022 Spotlight: Wasserstein Iterative Networks for Barycenter Estimation »
Alexander Korotin · Vage Egiazarian · Lingxiao Li · Evgeny Burnaev -
2021 Poster: Manifold Topology Divergence: a Framework for Comparing Data Manifolds. »
Serguei Barannikov · Ilya Trofimov · Grigorii Sotnikov · Ekaterina Trimbach · Alexander Korotin · Alexander Filippov · Evgeny Burnaev -
2021 Poster: BooVAE: Boosting Approach for Continual Learning of VAE »
Evgenii Egorov · Anna Kuzina · Evgeny Burnaev -
2021 Poster: Object DGCNN: 3D Object Detection using Dynamic Graphs »
Yue Wang · Justin Solomon -
2021 Poster: Large-Scale Wasserstein Gradient Flows »
Petr Mokrov · Alexander Korotin · Lingxiao Li · Aude Genevay · Justin Solomon · Evgeny Burnaev -
2021 Poster: MarioNette: Self-Supervised Sprite Learning »
Dmitriy Smirnov · MICHAEL GHARBI · Matthew Fisher · Vitor Guizilini · Alexei Efros · Justin Solomon -
2020 Poster: Continuous Regularized Wasserstein Barycenters »
Lingxiao Li · Aude Genevay · Mikhail Yurochkin · Justin Solomon -
2019 : Poster Session »
Lili Yu · Aleksei Kroshnin · Alex Delalande · Andrew Carr · Anthony Tompkins · Aram-Alexandre Pooladian · Arnaud Robert · Ashok Vardhan Makkuva · Aude Genevay · Bangjie Liu · Bo Zeng · Charlie Frogner · Elsa Cazelles · Esteban G Tabak · Fabio Ramos · François-Pierre PATY · Georgios Balikas · Giulio Trigila · Hao Wang · Hinrich Mahler · Jared Nielsen · Karim Lounici · Kyle Swanson · Mukul Bhutani · Pierre Bréchet · Piotr Indyk · samuel cohen · Stefanie Jegelka · Tao Wu · Thibault Sejourne · Tudor Manole · Wenjun Zhao · Wenlin Wang · Wenqi Wang · Yonatan Dukler · Zihao Wang · Chaosheng Dong -
2019 : Aude Genevay »
Aude Genevay -
2019 Poster: PRNet: Self-Supervised Learning for Partial-to-Partial Registration »
Yue Wang · Justin Solomon -
2019 Poster: Alleviating Label Switching with Optimal Transport »
Pierre Monteiller · Sebastian Claici · Edward Chien · Farzaneh Mirzazadeh · Justin Solomon · Mikhail Yurochkin -
2019 Poster: Hierarchical Optimal Transport for Document Representation »
Mikhail Yurochkin · Sebastian Claici · Edward Chien · Farzaneh Mirzazadeh · Justin Solomon -
2017 Poster: Parallel Streaming Wasserstein Barycenters »
Matt Staib · Sebastian Claici · Justin Solomon · Stefanie Jegelka -
2017 Tutorial: A Primer on Optimal Transport »
Marco Cuturi · Justin Solomon