Timezone: »
Poster
Linear Convergence with Condition Number Independent Access of Full Gradients
Lijun Zhang · Mehrdad Mahdavi · Rong Jin
Thu Dec 05 07:00 PM -- 11:59 PM (PST) @ Harrah's Special Events Center, 2nd Floor #None
For smooth and strongly convex optimization, the optimal iteration complexity of the gradient-based algorithm is $O(\sqrt{\kappa}\log 1/\epsilon)$, where $\kappa$ is the conditional number. In the case that the optimization problem is ill-conditioned, we need to evaluate a larger number of full gradients, which could be computationally expensive. In this paper, we propose to reduce the number of full gradient required by allowing the algorithm to access the stochastic gradients of the objective function. To this end, we present a novel algorithm named Epoch Mixed Gradient Descent (EMGD) that is able to utilize two kinds of gradients. A distinctive step in EMGD is the mixed gradient descent, where we use an combination of the gradient and the stochastic gradient to update the intermediate solutions. By performing a fixed number of mixed gradient descents, we are able to improve the sub-optimality of the solution by a constant factor, and thus achieve a linear convergence rate. Theoretical analysis shows that EMGD is able to find an $\epsilon$-optimal solution by computing $O(\log 1/\epsilon)$ full gradients and $O(\kappa^2\log 1/\epsilon)$ stochastic gradients.
Author Information
Lijun Zhang (Nanjing University (NJU))
Mehrdad Mahdavi (Michigan State University (MSU))
Rong Jin (Michigan State University (MSU))
More from the Same Authors
-
2020 Poster: Dynamic Regret of Convex and Smooth Functions »
Peng Zhao · Yu-Jie Zhang · Lijun Zhang · Zhi-Hua Zhou -
2018 Poster: Adaptive Online Learning in Dynamic Environments »
Lijun Zhang · Shiyin Lu · Zhi-Hua Zhou -
2018 Poster: $\ell_1$-regression with Heavy-tailed Distributions »
Lijun Zhang · Zhi-Hua Zhou -
2018 Poster: Fast Rates of ERM and Stochastic Approximation: Adaptive to Error Bound Conditions »
Mingrui Liu · Xiaoxuan Zhang · Lijun Zhang · Rong Jin · Tianbao Yang -
2017 Poster: Scalable Demand-Aware Recommendation »
Jinfeng Yi · Cho-Jui Hsieh · Kush Varshney · Lijun Zhang · Yao Li -
2017 Poster: Improved Dynamic Regret for Non-degenerate Functions »
Lijun Zhang · Tianbao Yang · Jinfeng Yi · Rong Jin · Zhi-Hua Zhou -
2017 Poster: Learning with Feature Evolvable Streams »
Bo-Jian Hou · Lijun Zhang · Zhi-Hua Zhou -
2014 Poster: Extracting Certainty from Uncertainty: Transductive Pairwise Classification from Pairwise Similarities »
Tianbao Yang · Rong Jin -
2014 Poster: Top Rank Optimization in Linear Time »
Nan Li · Rong Jin · Zhi-Hua Zhou -
2013 Poster: Mixed Optimization for Smooth Functions »
Mehrdad Mahdavi · Lijun Zhang · Rong Jin -
2013 Poster: Stochastic Convex Optimization with Multiple Objectives »
Mehrdad Mahdavi · Tianbao Yang · Rong Jin -
2013 Poster: Speedup Matrix Completion with Side Information: Application to Multi-Label Learning »
Miao Xu · Rong Jin · Zhi-Hua Zhou -
2012 Poster: Nystr{รถ}m Method vs Random Fourier Features: A Theoretical and Empirical Comparison »
Tianbao Yang · Yu-Feng Li · Mehrdad Mahdavi · Rong Jin · Zhi-Hua Zhou -
2012 Poster: Semi-Crowdsourced Clustering: Generalizing Crowd Labeling by Robust Distance Metric Learning »
Jinfeng Yi · Rong Jin · Anil K Jain · Shaili Jain -
2012 Poster: Stochastic Gradient Descent with Only One Projection »
Mehrdad Mahdavi · Tianbao Yang · Rong Jin · Shenghuo Zhu -
2010 Poster: Active Learning by Querying Informative and Representative Examples »
Sheng-Jun Huang · Rong Jin · Zhi-Hua Zhou -
2010 Poster: Multi-label Multiple Kernel Learning by Stochastic Approximation: Application to Visual Object Recognition »
Serhat S Bucak · Rong Jin · Anil K Jain -
2009 Poster: Adaptive Regularization for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael Lyu · Zhirong Yang -
2009 Spotlight: Adaptive Regularization for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael Lyu · Zhirong Yang -
2009 Poster: Regularized Distance Metric Learning:Theory and Algorithm »
Rong Jin · Shijun Wang · Yang Zhou -
2009 Poster: Learning Bregman Distance Functions and Its Application for Semi-Supervised Clustering »
Lei Wu · Rong Jin · Steven Chu-Hong Hoi · Jianke Zhu · Nenghai Yu -
2009 Poster: DUOL: A Double Updating Approach for Online Learning »
Peilin Zhao · Steven Chu-Hong Hoi · Rong Jin -
2009 Poster: Learning to Rank by Optimizing NDCG Measure »
Hamed Valizadegan · Rong Jin · Ruofei Zhang · Jianchang Mao -
2009 Spotlight: Learning to Rank by Optimizing NDCG Measure »
Hamed Valizadegan · Rong Jin · Ruofei Zhang · Jianchang Mao -
2008 Poster: Multi-label Multiple Kernel Learning »
Shuiwang Ji · Liang Sun · Rong Jin · Jieping Ye -
2008 Spotlight: Multi-label Multiple Kernel Learning »
Shuiwang Ji · Liang Sun · Rong Jin · Jieping Ye -
2008 Poster: An Extended Level Method for Efficient Multiple Kernel Learning »
Zenglin Xu · Rong Jin · Irwin King · Michael Lyu -
2008 Poster: Semi-supervised Learning with Weakly-Related Unlabeled Data : Towards Better Text Categorization »
Liu Yang · Rong Jin · Rahul Sukthankar -
2008 Spotlight: Semi-supervised Learning with Weakly-Related Unlabeled Data : Towards Better Text Categorization »
Liu Yang · Rong Jin · Rahul Sukthankar -
2007 Poster: Efficient Convex Relaxation for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael Lyu -
2006 Poster: Generalized Maximum Margin Clustering and Unsupervised Kernel Learning »
Hamed Valizadegan · Rong Jin