Timezone: »
Automatic relevance determination (ARD), and the closely-related sparse Bayesian learning (SBL) framework, are effective tools for pruning large numbers of irrelevant features. However, popular update rules used for this process are either prohibitively slow in practice and/or heuristic in nature without proven convergence properties. This paper furnishes an alternative means of optimizing a general ARD cost function using an auxiliary function that can naturally be solved using a series of re-weighted L1 problems. The result is an efficient algorithm that can be implemented using standard convex programming toolboxes and is guaranteed to converge to a stationary point unlike existing methods. The analysis also leads to additional insights into the behavior of previous ARD updates as well as the ARD cost function. For example, the standard fixed-point updates of MacKay (1992) are shown to be iteratively solving a particular min-max problem, although they are not guaranteed to lead to a stationary point. The analysis also reveals that ARD is exactly equivalent to performing MAP estimation using a particular feature- and noise-dependent \textit{non-factorial} weight prior with several desirable properties over conventional priors with respect to feature selection. In particular, it provides a tighter approximation to the L0 quasi-norm sparsity measure than the L1 norm. Overall these results suggests alternative cost functions and update procedures for selecting features and promoting sparse solutions.
Author Information
David P Wipf (AWS)
Srikantan Nagarajan
More from the Same Authors
-
2021 : A Closer Look at Distribution Shifts and Out-of-Distribution Generalization on Graphs »
Mucong Ding · Kezhi Kong · Jiuhai Chen · John Kirchenbauer · Micah Goldblum · David P Wipf · Furong Huang · Tom Goldstein -
2022 Poster: Learning Enhanced Representation for Tabular Data via Neighborhood Propagation »
Kounianhua Du · Weinan Zhang · Ruiwen Zhou · Yangkun Wang · Xilong Zhao · Jiarui Jin · Quan Gan · Zheng Zhang · David P Wipf -
2022 Spotlight: Lightning Talks 5B-3 »
Yanze Wu · Jie Xiao · Nianzu Yang · Jieyi Bi · Jian Yao · Yiting Chen · Qizhou Wang · Yangru Huang · Yongqiang Chen · Peixi Peng · Yuxin Hong · Xintao Wang · Feng Liu · Yining Ma · Qibing Ren · Xueyang Fu · Yonggang Zhang · Kaipeng Zeng · Jiahai Wang · GEN LI · Yonggang Zhang · Qitian Wu · Yifan Zhao · Chiyu Wang · Junchi Yan · Feng Wu · Yatao Bian · Xiaosong Jia · Ying Shan · Zhiguang Cao · Zheng-Jun Zha · Guangyao Chen · Tianjun Xiao · Han Yang · Jing Zhang · Jinbiao Chen · MA Kaili · Yonghong Tian · Junchi Yan · Chen Gong · Tong He · Binghui Xie · Yuan Sun · Francesco Locatello · Tongliang Liu · Yeow Meng Chee · David P Wipf · Tongliang Liu · Bo Han · Bo Han · Yanwei Fu · James Cheng · Zheng Zhang -
2022 Spotlight: Self-supervised Amodal Video Object Segmentation »
Jian Yao · Yuxin Hong · Chiyu Wang · Tianjun Xiao · Tong He · Francesco Locatello · David P Wipf · Yanwei Fu · Zheng Zhang -
2022 Spotlight: NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification »
Qitian Wu · Wentao Zhao · Zenan Li · David P Wipf · Junchi Yan -
2022 Spotlight: Lightning Talks 1B-1 »
Qitian Wu · Runlin Lei · Rongqin Chen · Luca Pinchetti · Yangze Zhou · Abhinav Kumar · Hans Hao-Hsun Hsu · Wentao Zhao · Chenhao Tan · Zhen Wang · Shenghui Zhang · Yuesong Shen · Tommaso Salvatori · Gitta Kutyniok · Zenan Li · Amit Sharma · Leong Hou U · Yordan Yordanov · Christian Tomani · Bruno Ribeiro · Yaliang Li · David P Wipf · Daniel Cremers · Bolin Ding · Beren Millidge · Ye Li · Yuhang Song · Junchi Yan · Zhewei Wei · Thomas Lukasiewicz -
2022 Poster: NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification »
Qitian Wu · Wentao Zhao · Zenan Li · David P Wipf · Junchi Yan -
2022 Poster: Transformers from an Optimization Perspective »
Yongyi Yang · zengfeng Huang · David P Wipf -
2022 Poster: Descent Steps of a Relation-Aware Energy Produce Heterogeneous Graph Neural Networks »
Hongjoon Ahn · Yongyi Yang · Quan Gan · Taesup Moon · David P Wipf -
2022 Poster: Self-supervised Amodal Video Object Segmentation »
Jian Yao · Yuxin Hong · Chiyu Wang · Tianjun Xiao · Tong He · Francesco Locatello · David P Wipf · Yanwei Fu · Zheng Zhang -
2022 Poster: Learning Manifold Dimensions with Conditional Variational Autoencoders »
Yijia Zheng · Tong He · Yixuan Qiu · David P Wipf -
2021 : A Closer Look at Distribution Shifts and Out-of-Distribution Generalization on Graphs »
Mucong Ding · Kezhi Kong · Jiuhai Chen · John Kirchenbauer · Micah Goldblum · David P Wipf · Furong Huang · Tom Goldstein -
2020 Poster: Further Analysis of Outlier Detection with Deep Generative Models »
Ziyu Wang · Bin Dai · David P Wipf · Jun Zhu -
2012 Poster: Dual-Space Analysis of the Sparse Linear Model »
David P Wipf -
2011 Poster: Sparse Estimation with Structured Dictionaries »
David P Wipf -
2011 Spotlight: Sparse Estimation with Structured Dictionaries »
David P Wipf -
2009 Poster: Sparse Estimation Using General Likelihoods and Non-Factorial Priors »
David P Wipf · Sri Nagarajan -
2008 Poster: Estimating the Location and Orientation of Complex, Correlated Neural Activity using MEG »
David P Wipf · Julia Owen · Hagai Attias · Kensuke Sekihara · Sri Nagarajan -
2008 Spotlight: Estimating the Location and Orientation of Complex, Correlated Neural Activity using MEG »
David P Wipf · Julia Owen · Hagai Attias · Kensuke Sekihara · Sri Nagarajan -
2006 Poster: Analysis of Empirical Bayesian Methods for Neuroelectromagnetic Source Localization »
David P Wipf · Rey R Ramirez · Jason A Palmer · Scott Makeig · Bhaskar Rao -
2006 Spotlight: Analysis of Empirical Bayesian Methods for Neuroelectromagnetic Source Localization »
David P Wipf · Rey R Ramirez · Jason A Palmer · Scott Makeig · Bhaskar Rao -
2006 Poster: A Probabilistic Algorithm Integrating Source Localization and Noise Suppression for MEG and EEG data »
Johanna M Zumer · Hagai Attias · Kensuke Sekihara · Srikantan Nagarajan