Timezone: »
Poster
On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
Changyou Chen · Nan Ding · Lawrence Carin
Recent advances in Bayesian learning with large-scale data have witnessed emergence of stochastic gradient MCMC algorithms (SG-MCMC), such as stochastic gradient Langevin dynamics (SGLD), stochastic gradient Hamiltonian MCMC (SGHMC), and the stochastic gradient thermostat. While finite-time convergence properties of the SGLD with a 1st-order Euler integrator have recently been studied, corresponding theory for general SG-MCMCs has not been explored. In this paper we consider general SG-MCMCs with high-order integrators, and develop theory to analyze finite-time convergence properties and their asymptotic invariant measures. Our theoretical results show faster convergence rates and more accurate invariant measures for SG-MCMCs with higher-order integrators. For example, with the proposed efficient 2nd-order symmetric splitting integrator, the mean square error (MSE) of the posterior average for the SGHMC achieves an optimal convergence rate of $L^{-4/5}$ at $L$ iterations, compared to $L^{-2/3}$ for the SGHMC and SGLD with 1st-order Euler integrators. Furthermore, convergence results of decreasing-step-size SG-MCMCs are also developed, with the same convergence rates as their fixed-step-size counterparts for a specific decreasing sequence. Experiments on both synthetic and real datasets verify our theory, and show advantages of the proposed method in two large-scale real applications.
Author Information
Changyou Chen (Duke University)
Nan Ding (Google)
Lawrence Carin (Duke University)
More from the Same Authors
-
2021 Spotlight: Supercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer »
Junya Chen · Zidi Xiu · Benjamin Goldstein · Ricardo Henao · Lawrence Carin · Chenyang Tao -
2022 : CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks »
Sakshi Varshney · Vinay Verma · Srijith PK · Piyush Rai · Lawrence Carin -
2022 Poster: Why do We Need Large Batchsizes in Contrastive Learning? A Gradient-Bias Perspective »
Changyou Chen · Jianyi Zhang · Yi Xu · Liqun Chen · Jiali Duan · Yiran Chen · Son Tran · Belinda Zeng · Trishul Chilimbi -
2021 Poster: Supercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer »
Junya Chen · Zidi Xiu · Benjamin Goldstein · Ricardo Henao · Lawrence Carin · Chenyang Tao -
2021 Poster: CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks »
Sakshi Varshney · Vinay Kumar Verma · P. K. Srijith · Lawrence Carin · Piyush Rai -
2021 Poster: Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning »
Nan Ding · Xi Chen · Tomer Levinboim · Sebastian Goodman · Radu Soricut -
2020 Poster: Learning Manifold Implicitly via Explicit Heat-Kernel Learning »
Yufan Zhou · Changyou Chen · Jinhui Xu -
2020 Poster: GAN Memory with No Forgetting »
Yulai Cong · Miaoyun Zhao · Jianqiao Li · Sijia Wang · Lawrence Carin -
2020 Poster: Reconsidering Generative Objectives For Counterfactual Reasoning »
Danni Lu · Chenyang Tao · Junya Chen · Fan Li · Feng Guo · Lawrence Carin -
2020 Poster: AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning »
Hao Zhang · Yuan Li · Zhijie Deng · Xiaodan Liang · Lawrence Carin · Eric Xing -
2020 Poster: Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability »
Nathan Inkawhich · Kevin J Liang · Binghui Wang · Matthew Inkawhich · Lawrence Carin · Yiran Chen -
2020 Poster: Calibrating CNNs for Lifelong Learning »
Pravendra Singh · Vinay Kumar Verma · Pratik Mazumder · Lawrence Carin · Piyush Rai -
2020 Poster: Bayesian Multi-type Mean Field Multi-agent Imitation Learning »
Fan Yang · Alina Vereshchaka · Changyou Chen · Wen Dong -
2020 Spotlight: Bayesian Multi-type Mean Field Multi-agent Imitation Learning »
Fan Yang · Alina Vereshchaka · Changyou Chen · Wen Dong -
2019 Poster: Improving Textual Network Learning with Variational Homophilic Embeddings »
Wenlin Wang · Chenyang Tao · Zhe Gan · Guoyin Wang · Liqun Chen · Xinyuan Zhang · Ruiyi Zhang · Qian Yang · Ricardo Henao · Lawrence Carin -
2019 Poster: Ouroboros: On Accelerating Training of Transformer-Based Language Models »
Qian Yang · Zhouyuan Huo · Wenlin Wang · Lawrence Carin -
2019 Poster: Scalable Gromov-Wasserstein Learning for Graph Partitioning and Matching »
Hongteng Xu · Dixin Luo · Lawrence Carin -
2019 Poster: Kernel-Based Approaches for Sequence Modeling: Connections to Neural Methods »
Kevin J Liang · Guoyin Wang · Yitong Li · Ricardo Henao · Lawrence Carin -
2019 Poster: Certified Adversarial Robustness with Additive Noise »
Bai Li · Changyou Chen · Wenlin Wang · Lawrence Carin -
2019 Poster: On Fenchel Mini-Max Learning »
Chenyang Tao · Liqun Chen · Shuyang Dai · Junya Chen · Ke Bai · Dong Wang · Jianfeng Feng · Wenlian Lu · Georgiy Bobashev · Lawrence Carin -
2019 Poster: Reward Constrained Interactive Recommendation with Natural Language Feedback »
Ruiyi Zhang · Tong Yu · Yilin Shen · Hongxia Jin · Changyou Chen -
2018 Poster: Adversarial Text Generation via Feature-Mover's Distance »
Liqun Chen · Shuyang Dai · Chenyang Tao · Haichao Zhang · Zhe Gan · Dinghan Shen · Yizhe Zhang · Guoyin Wang · Dinghan Shen · Lawrence Carin -
2018 Poster: Distilled Wasserstein Learning for Word Embedding and Topic Modeling »
Hongteng Xu · Wenlin Wang · Wei Liu · Lawrence Carin -
2018 Poster: Diffusion Maps for Textual Network Embedding »
Xinyuan Zhang · Yitong Li · Dinghan Shen · Lawrence Carin -
2018 Spotlight: Diffusion Maps for Textual Network Embedding »
Xinyuan Zhang · Yitong Li · Dinghan Shen · Lawrence Carin -
2017 Spotlight: Targeting EEG/LFP Synchrony with Neural Nets »
Yitong Li · michael Murias · samantha Major · geraldine Dawson · Kafui Dzirasa · Lawrence Carin · David Carlson -
2017 Poster: Targeting EEG/LFP Synchrony with Neural Nets »
Yitong Li · michael Murias · samantha Major · geraldine Dawson · Kafui Dzirasa · Lawrence Carin · David Carlson -
2017 Poster: Triangle Generative Adversarial Networks »
Zhe Gan · Liqun Chen · Weiyao Wang · Yuchen Pu · Yizhe Zhang · Hao Liu · Chunyuan Li · Lawrence Carin -
2017 Poster: ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching »
Chunyuan Li · Hao Liu · Changyou Chen · Yuchen Pu · Liqun Chen · Ricardo Henao · Lawrence Carin -
2017 Poster: An inner-loop free solution to inverse problems using deep neural networks »
Kai Fan · Qi Wei · Lawrence Carin · Katherine Heller -
2017 Poster: Cold-Start Reinforcement Learning with Softmax Policy Gradient »
Nan Ding · Radu Soricut -
2017 Poster: VAE Learning via Stein Variational Gradient Descent »
Yuchen Pu · Zhe Gan · Ricardo Henao · Chunyuan Li · Shaobo Han · Lawrence Carin -
2017 Poster: Deconvolutional Paragraph Representation Learning »
Yizhe Zhang · Dinghan Shen · Guoyin Wang · Zhe Gan · Ricardo Henao · Lawrence Carin -
2017 Poster: Adversarial Symmetric Variational Autoencoder »
Yuchen Pu · Weiyao Wang · Ricardo Henao · Liqun Chen · Zhe Gan · Chunyuan Li · Lawrence Carin -
2017 Poster: A Probabilistic Framework for Nonlinearities in Stochastic Neural Networks »
Qinliang Su · xuejun Liao · Lawrence Carin -
2017 Poster: Scalable Model Selection for Belief Networks »
Zhao Song · Yusuke Muraoka · Ryohei Fujimaki · Lawrence Carin -
2017 Poster: Cross-Spectral Factor Analysis »
Neil Gallagher · Kyle Ulrich · Austin Talbot · Kafui Dzirasa · Lawrence Carin · David Carlson -
2016 Poster: Towards Unifying Hamiltonian Monte Carlo and Slice Sampling »
Yizhe Zhang · Xiangyu Wang · Changyou Chen · Ricardo Henao · Kai Fan · Lawrence Carin -
2016 Poster: Variational Autoencoder for Deep Learning of Images, Labels and Captions »
Yunchen Pu · Zhe Gan · Ricardo Henao · Xin Yuan · Chunyuan Li · Andrew Stevens · Lawrence Carin -
2016 Poster: Linear Feature Encoding for Reinforcement Learning »
Zhao Song · Ronald Parr · Xuejun Liao · Lawrence Carin -
2016 Poster: Stochastic Gradient MCMC with Stale Gradients »
Changyou Chen · Nan Ding · Chunyuan Li · Yizhe Zhang · Lawrence Carin -
2015 Poster: GP Kernels for Cross-Spectrum Analysis »
Kyle R Ulrich · David Carlson · Kafui Dzirasa · Lawrence Carin -
2015 Poster: Deep Poisson Factor Modeling »
Ricardo Henao · Zhe Gan · James Lu · Lawrence Carin -
2015 Poster: Preconditioned Spectral Descent for Deep Learning »
David Carlson · Edo Collins · Ya-Ping Hsieh · Lawrence Carin · Volkan Cevher -
2015 Poster: Large-Scale Bayesian Multi-Label Learning via Topic-Based Label Embeddings »
Piyush Rai · Changwei Hu · Ricardo Henao · Lawrence Carin -
2015 Spotlight: Large-Scale Bayesian Multi-Label Learning via Topic-Based Label Embeddings »
Piyush Rai · Changwei Hu · Ricardo Henao · Lawrence Carin -
2015 Poster: Deep Temporal Sigmoid Belief Networks for Sequence Modeling »
Zhe Gan · Chunyuan Li · Ricardo Henao · David Carlson · Lawrence Carin -
2015 Poster: Embedding Inference for Structured Multilabel Prediction »
Farzaneh Mirzazadeh · Siamak Ravanbakhsh · Nan Ding · Dale Schuurmans -
2014 Poster: Analysis of Brain States from Multi-Region LFP Time-Series »
Kyle R Ulrich · David Carlson · Wenzhao Lian · Jana S Borg · Kafui Dzirasa · Lawrence Carin -
2014 Poster: Bayesian Sampling Using Stochastic Gradient Thermostats »
Nan Ding · Youhan Fang · Ryan Babbush · Changyou Chen · Robert D Skeel · Hartmut Neven -
2014 Poster: Bayesian Nonlinear Support Vector Machines and Discriminative Factor Modeling »
Ricardo Henao · Xin Yuan · Lawrence Carin -
2014 Poster: Compressive Sensing of Signals from a GMM with Sparse Precision Matrices »
Jianbo Yang · Xuejun Liao · Minhua Chen · Lawrence Carin -
2014 Poster: On the relations of LFPs & Neural Spike Trains »
David Carlson · Jana Schaich Borg · Kafui Dzirasa · Lawrence Carin -
2014 Poster: Dynamic Rank Factor Model for Text Streams »
Shaobo Han · Lin Du · Esther Salazar · Lawrence Carin -
2014 Poster: Robust Bayesian Max-Margin Clustering »
Changyou Chen · Jun Zhu · Xinhua Zhang -
2013 Poster: Dynamic Clustering via Asymptotics of the Dependent Dirichlet Process Mixture »
Trevor Campbell · Miao Liu · Brian Kulis · Jonathan How · Lawrence Carin -
2013 Poster: Designed Measurements for Vector Count Data »
Liming Wang · David Carlson · Miguel Rodrigues · David Wilcox · Robert Calderbank · Lawrence Carin -
2013 Poster: Integrated Non-Factorized Variational Inference »
Shaobo Han · Xuejun Liao · Lawrence Carin -
2013 Poster: Real-Time Inference for a Gamma Process Model of Neural Spiking »
David Carlson · Vinayak Rao · Joshua T Vogelstein · Lawrence Carin -
2012 Workshop: Bayesian Nonparametric Models For Reliable Planning And Decision-Making Under Uncertainty »
Jonathan How · Lawrence Carin · John Fisher III · Michael Jordan · Alborz Geramifard -
2012 Poster: Joint Modeling of a Matrix with Associated Text via Latent Binary Features »
XianXing Zhang · Lawrence Carin -
2012 Poster: Augment-and-Conquer Negative Binomial Processes »
Mingyuan Zhou · Lawrence Carin -
2012 Spotlight: Augment-and-Conquer Negative Binomial Processes »
Mingyuan Zhou · Lawrence Carin -
2011 Poster: On the Analysis of Multi-Channel Neural Spike Data »
Bo Chen · David Carlson · Lawrence Carin -
2011 Poster: t-divergence Based Approximate Inference »
Nan Ding · S.V.N. Vishwanathan · Yuan Qi -
2011 Poster: The Kernel Beta Process »
Lu Ren · Yingjian Wang · David B Dunson · Lawrence Carin -
2011 Spotlight: The Kernel Beta Process »
Lu Ren · Yingjian Wang · David B Dunson · Lawrence Carin -
2011 Poster: Hierarchical Topic Modeling for Analysis of Time-Evolving Personal Choices »
XianXing Zhang · David B Dunson · Lawrence Carin -
2010 Poster: t-logistic regression »
Nan Ding · S.V.N. Vishwanathan -
2010 Poster: Joint Analysis of Time-Evolving Binary Matrices and Associated Documents »
Eric X Wang · Dehong Liu · Jorge G Silva · David B Dunson · Lawrence Carin -
2009 Poster: A Bayesian Model for Simultaneous Image Clustering, Annotation and Object Segmentation »
Lan Du · Lu Ren · David B Dunson · Lawrence Carin -
2009 Poster: Non-Parametric Bayesian Dictionary Learning for Sparse Image Representations »
Mingyuan Zhou · Haojun Chen · John Paisley · Lu Ren · Guillermo Sapiro · Lawrence Carin -
2009 Poster: Learning to Explore and Exploit in POMDPs »
Chenghui Cai · Xuejun Liao · Lawrence Carin -
2008 Workshop: Cost Sensitive Learning »
Balaji R Krishnapuram · Shipeng Yu · Oksana Yakhnenko · R. Bharat Rao · Lawrence Carin -
2007 Poster: Semi-Supervised Multitask Learning »
Qiuhua Liu · Xuejun Liao · Lawrence Carin -
2007 Spotlight: Semi-Supervised Multitask Learning »
Qiuhua Liu · Xuejun Liao · Lawrence Carin