Timezone: »
Poster
A Universal Approximation Theorem of Deep Neural Networks for Expressing Probability Distributions
Yulong Lu · Jianfeng Lu
This paper studies the universal approximation property of deep neural networks for representing probability distributions. Given a target distribution $\pi$ and a source distribution $p_z$ both defined on $\mathbb{R}^d$, we prove under some assumptions that there exists a deep neural network $g:\mathbb{R}^d\gt \mathbb{R}$ with ReLU activation such that the pushforward measure $(\nabla g)_\# p_z$ of $p_z$ under the map $\nabla g$ is arbitrarily close to the target measure $\pi$. The closeness are measured by three classes of integral probability metrics between probability distributions: $1$Wasserstein distance, maximum mean distance (MMD) and kernelized Stein discrepancy (KSD). We prove upper bounds for the size (width and depth) of the deep neural network in terms of the dimension $d$ and the approximation error $\varepsilon$ with respect to the three discrepancies. In particular, the size of neural network can grow exponentially in $d$ when $1$Wasserstein distance is used as the discrepancy, whereas for both MMD and KSD the size of neural network only depends on $d$ at most polynomially. Our proof relies on convergence estimates of empirical measures under aforementioned discrepancies and semidiscrete optimal transport.
Author Information
Yulong Lu (University of Massachusetts Amherst)
Jianfeng Lu (Duke University)
More from the Same Authors

2021 Spotlight: On the Representation of Solutions to Elliptic PDEs in Barron Spaces »
Ziang Chen · Jianfeng Lu · Yulong Lu 
2021 : Statistical Numerical PDE : Fast Rate, Neural Scaling Law and When itâ€™s Optimal »
Yiping Lu · Haoxuan Chen · Jianfeng Lu · Lexing Ying · Jose Blanchet 
2021 Poster: On the Representation of Solutions to Elliptic PDEs in Barron Spaces »
Ziang Chen · Jianfeng Lu · Yulong Lu