Timezone: »
Dear NIPS Workshop Chairs,
We propose to organize the workshop
OPT2011 "Optimization for Machine Learning."
This workshop builds on precedent established by our previously very wellreceived NIPS workshops, OPT2008OPT2010 (Urls are cited in the last box)
The OPT workshops enjoyed packed (to overpacked) attendanceand this enthusiastic reception underscores the strong interest, relevance, and importance enjoyed by optimization in the ML community.
This continued interest in optimization is readily acknowledged, because optimization lies at the heart of ML algorithms. Sometimes, classical textbook algorithms suffice, but the majority problems require tailored methods that
are based on a deeper understanding of the ML requirements. In fact, ML applications and researchers are driving some of the most cuttingedge developments in optimization today. The intimate relation of optimization with ML is the key motivation for our workshop, which aims to foster discussion,
discovery, and dissemination of the stateoftheart in optimization.
FURTHER DETAILS

Optimization is indispensable to many machine learning algorithms. What can we say beyond this obvious realization?
Previous talks at the OPT workshops have covered frameworks for convex programs (D. Bertsekas), the intersection of ML and optimization, especially in the area of SVM training (S. Wright), largescale learning via stochastic
gradient methods and its tradeoffs (L. Bottou, N. Srebro), exploitation of structured sparsity in optimization (Vandenberghe), randomized methods for extremely largescale convex optimization (A. Nemirovski), and complexity theoretic foundations of convex optimization (Y. Nesterov), among others.
Several important realizations were brought to the fore by these talks, and many of the dominant ideas will appear in our forthcoming book: "Optimization for Machine learning" (MIT Press, 2011).
Much interest has focused recently on stochastic methods, which can be used in an online setting and in settings where data sets are extremely large and high accuracy is not required. Many aspects of stochastic gradient remain to be
explored, for example, different algorithmic variants, customizing to the data set structure, convergence analysis, sampling techniques, software, choice of regularization and tradeoff parameters, distributed and parallel computation. The need for an uptodate analysis of algorithms for nonconvex
problems remains an important practical issue, whose importance becomes even more pronounced as ML tackles more and more complex mathematical models.
Finally, we do not wish to ignore the "not particularly large scale" setting, where one does have time to wield substantial computational resources. In this setting, highaccuracy solutions and deep understanding of the lessons contained in the data are needed. Examples valuable to MLers may be
exploration of genetic and environmental data to identify risk factors for disease; or problems dealing with setups where the amount of observed data is not huge, but the mathematical model is complex.
PRELIMINARY CFP (which will be circulated) FOLLOWS

OPT 2011
(proposed) NIPS Workshop on Optimization for Machine Learning
NIPS2011 Workshop
URL: http://opt.kyb.tuebingen.mpg.de/index.html

Abstract

Optimization is a wellestablished, mature discipline. But the way we use this discipline is undergoing a rapid transformation: the advent of modern data intensive applications in statistics, scientific computing, or data mining and machine learning, is forcing us to drop theoretically powerful methods in favor of simpler but more scalable ones. This changeover exhibits itself most starkly in machine learning, where we have to often process massive datasets;
this necessitates not only reliance on largescale optimization techniques, but also the need to develop methods "tuned" to the specific needs of machine learning problems.
Background and Objectives

We build on OPT2008, OPT2009, and OPT2010the forerunners of this workshop. All three workshops happened as a part of NIPS. Beyond this major precedent, there have been other related workshops such as the "Mathematical
Programming in Machine Learning / Data Mining" series (2005 to 2007) and the BigML NIPS 2007 workshop.
Our workshop has the following major aims:
* Provide a platform for increasing the interaction between researchers from optimization, operations research, statistics, scientific computing, and machine learning;
* Identify key problems and challenges that lie at the intersection of optimization and ML;
* Narrow the gap between optimization and ML, to help reduce rediscovery, and thereby accelerate new advances.
Call for Participation

This year we invite two types of submissions to the workshop:
(i) contributed talks and/or posters
(ii) open problems
For the latter, we request the authors to prepare a few slides that clearly
present, motivate, and explain an important open problem  the main aim here
is to foster active discussion. Our call for open problems is modeled after a
similar session that takes place at COLT. The topics of interest for the open
problem session are the same as those for regular submissions; please see
below for details.
In addition to open problems, we invite high quality submissions for
presentation as talks or poster presentations during the workshop. We are
especially interested in participants who can contribute theory / algorithms,
applications, or implementations with a machine learning focus on the
following topics:
Topics

* Stochastic, Parallel and Online Optimization,
 Largescale learning, massive data sets
 Distributed algorithms
 Optimization on massively parallel architectures
 Optimization using GPUs, Streaming algorithms
 Decomposition for largescale, messagepassing and online learning
 Stochastic approximation
 Randomized algorithms
* Algorithms and Techniques (application oriented)
 Global and Lipschitz optimization
 Algorithms for nonsmooth optimization
 Linear and higherorder relaxations
 Polyhedral combinatorics applications to ML problems
* Nonconvex Optimization,
 Nonconvex quadratic programming, including binary QPs
 Convex Concave Decompositions, D.C. Programming, EM
 Training of deep architectures and large hidden variable models
 Approximation Algorithms
 Nonconvex, nonsmooth optimization
* Optimization with Sparsity constraints
 Combinatorial methods for L0 norm minimization
 L1, Lasso, Group Lasso, sparse PCA, sparse Gaussians
 Rank minimization methods
 Feature and subspace selection
* Combinatorial Optimization
 Optimization in Graphical Models
 Structure learning
 MAP estimation in continuous and discrete random fields
 Clustering and graphpartitioning
 Semisupervised and multipleinstance learning
Important Dates

* Deadline for submission of papers: 21st October 2011
* Notification of acceptance: 12th November 2011
* Final version of submission: 24th November 2011
Please note that at least one author of each accepted paper must be available
to present the paper at the workshop. Further details regarding the
submission process are available at the workshop homepage.
Workshop

The workshop will be a oneday event with a morning and afternoon session. In
addition to a lunch break, long coffee breaks will be offered both in the
morning and afternoon.
A new session on open problems is proposed for spurring active discussion and
interaction amongst the participants. A key aim of this session will be on
establishing areas and identifying problems of interest to the community.
Invited Speakers

Stephen Boyd (Stanford)
* Aharon BenTal (Technion)
* Ben Recht (UW Madison)
Workshop Organizers

* Suvrit Sra, Max Planck Institute for Intelligent Systems
* Sebastian Nowozin, Microsoft Research, Cambridge, UK
* Stephen Wright, University of Wisconsin, Madison

Author Information
Suvrit Sra (MIT)
Suvrit Sra is a faculty member within the EECS department at MIT, where he is also a core faculty member of IDSS, LIDS, MITML Group, as well as the statistics and data science center. His research spans topics in optimization, matrix theory, differential geometry, and probability theory, which he connects with machine learning  a key focus of his research is on the theme "Optimization for Machine Learning” (http://optml.org)
Stephen Wright (UWMadison)
Steve Wright is a Professor of Computer Sciences at the University of WisconsinMadison. His research interests lie in computational optimization and its applications to science and engineering. Prior to joining UWMadison in 2001, Wright was a Senior Computer Scientist (19972001) and Computer Scientist (19901997) at Argonne National Laboratory, and Professor of Computer Science at the University of Chicago (20002001). He is the past Chair of the Mathematical Optimization Society (formerly the Mathematical Programming Society), the leading professional society in optimization, and a member of the Board of the Society for Industrial and Applied Mathematics (SIAM). Wright is the author or coauthor of four widely used books in numerical optimization, including "Primal Dual InteriorPoint Methods" (SIAM, 1997) and "Numerical Optimization" (with J. Nocedal, Second Edition, Springer, 2006). He has also authored over 85 refereed journal papers on optimization theory, algorithms, software, and applications. He is coauthor of widely used interiorpoint software for linear and quadratic optimization. His recent research includes algorithms, applications, and theory for sparse optimization (including applications in compressed sensing and machine learning).
Sebastian Nowozin (Microsoft Research)
More from the Same Authors

2021 Poster: Can contrastive learning avoid shortcut solutions? »
Joshua Robinson · Li Sun · Ke Yu · Kayhan Batmanghelich · Stefanie Jegelka · Suvrit Sra 
2021 Poster: Three Operator Splitting with Subgradients, Stochastic Gradients, and Adaptive Learning Rates »
Alp Yurtsever · Alex Gu · Suvrit Sra 
2020 : Invited speaker: SGD without replacement: optimal rate analysis and more, Suvrit Sra »
Suvrit Sra 
2020 Poster: SGD with shuffling: optimal rates without component convexity and large epoch requirements »
Kwangjun Ahn · Chulhee Yun · Suvrit Sra 
2020 Spotlight: SGD with shuffling: optimal rates without component convexity and large epoch requirements »
Kwangjun Ahn · Chulhee Yun · Suvrit Sra 
2020 Poster: Why are Adaptive Methods Good for Attention Models? »
Jingzhao Zhang · Sai Praneeth Karimireddy · Andreas Veit · Seungyeon Kim · Sashank Reddi · Sanjiv Kumar · Suvrit Sra 
2020 Oral: Hogwild!: A LockFree Approach to Parallelizing Stochastic Gradient Descent »
Benjamin Recht · Christopher Ré · Stephen Wright · Feng Niu 
2020 Poster: Towards Minimax Optimal Reinforcement Learning in Factored Markov Decision Processes »
Yi Tian · Jian Qian · Suvrit Sra 
2020 Spotlight: Towards Minimax Optimal Reinforcement Learning in Factored Markov Decision Processes »
Yi Tian · Jian Qian · Suvrit Sra 
2019 : Secondorder methods for nonconvex optimization with complexity guarantees »
Stephen Wright 
2019 Poster: Flexible Modeling of Diversity with Strongly LogConcave Distributions »
Joshua Robinson · Suvrit Sra · Stefanie Jegelka 
2019 Poster: Are deep ResNets provably better than linear predictors? »
Chulhee Yun · Suvrit Sra · Ali Jadbabaie 
2019 Poster: Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity »
Chulhee Yun · Suvrit Sra · Ali Jadbabaie 
2019 Spotlight: Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity »
Chulhee Yun · Suvrit Sra · Ali Jadbabaie 
2018 Workshop: Smooth Games Optimization and Machine Learning »
Simon LacosteJulien · Ioannis Mitliagkas · Gauthier Gidel · Vasilis Syrgkanis · Eva Tardos · Leon Bottou · Sebastian Nowozin 
2018 Poster: Direct RungeKutta Discretization Achieves Acceleration »
Jingzhao Zhang · Aryan Mokhtari · Suvrit Sra · Ali Jadbabaie 
2018 Spotlight: Direct RungeKutta Discretization Achieves Acceleration »
Jingzhao Zhang · Aryan Mokhtari · Suvrit Sra · Ali Jadbabaie 
2018 Poster: Exponentiated Strongly Rayleigh Distributions »
Zelda Mariet · Suvrit Sra · Stefanie Jegelka 
2018 Poster: ATOMO: Communicationefficient Learning via Atomic Sparsification »
Hongyi Wang · Scott Sievert · Shengchao Liu · Zachary Charles · Dimitris Papailiopoulos · Stephen Wright 
2018 Tutorial: Negative Dependence, Stable Polynomials, and All That »
Suvrit Sra · Stefanie Jegelka 
2017 Workshop: OPT 2017: Optimization for Machine Learning »
Suvrit Sra · Sashank J. Reddi · Alekh Agarwal · Benjamin Recht 
2017 Poster: The Numerics of GANs »
Lars Mescheder · Sebastian Nowozin · Andreas Geiger 
2017 Spotlight: The Numerics of GANs »
Lars Mescheder · Sebastian Nowozin · Andreas Geiger 
2017 Poster: Elementary Symmetric Polynomials for Optimal Experimental Design »
Zelda Mariet · Suvrit Sra 
2017 Poster: kSupport and Ordered Weighted Sparsity for Overlapping Groups: Hardness and Algorithms »
Cong Han Lim · Stephen Wright 
2017 Poster: Stabilizing Training of Generative Adversarial Networks through Regularization »
Kevin Roth · Aurelien Lucchi · Sebastian Nowozin · Thomas Hofmann 
2017 Poster: Polynomial time algorithms for dual volume sampling »
Chengtao Li · Stefanie Jegelka · Suvrit Sra 
2016 Workshop: OPT 2016: Optimization for Machine Learning »
Suvrit Sra · Francis Bach · Sashank J. Reddi · Niao He 
2016 : Discussion panel »
Ian Goodfellow · Soumith Chintala · Arthur Gretton · Sebastian Nowozin · Aaron Courville · Yann LeCun · Emily Denton 
2016 : Taming nonconvexity via geometry »
Suvrit Sra 
2016 : Training Generative Neural Samplers using Variational Divergence »
Sebastian Nowozin 
2016 Poster: Fast Mixing Markov Chains for Strongly Rayleigh Measures, DPPs, and Constrained Sampling »
Chengtao Li · Suvrit Sra · Stefanie Jegelka 
2016 Poster: Kronecker Determinantal Point Processes »
Zelda Mariet · Suvrit Sra 
2016 Poster: fGAN: Training Generative Neural Samplers using Variational Divergence Minimization »
Sebastian Nowozin · Botond Cseke · Ryota Tomioka 
2016 Poster: Proximal Stochastic Methods for Nonsmooth Nonconvex FiniteSum Optimization »
Sashank J. Reddi · Suvrit Sra · Barnabas Poczos · Alexander Smola 
2016 Poster: Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds »
Hongyi Zhang · Sashank J. Reddi · Suvrit Sra 
2016 Poster: DISCO Nets : DISsimilarity COefficients Networks »
Diane Bouchacourt · Pawan K Mudigonda · Sebastian Nowozin 
2016 Tutorial: LargeScale Optimization: Beyond Stochastic Gradient Descent and Convexity »
Suvrit Sra · Francis Bach 
2015 Workshop: Optimization for Machine Learning (OPT2015) »
Suvrit Sra · Alekh Agarwal · Leon Bottou · Sashank J. Reddi 
2015 Poster: Matrix Manifold Optimization for Gaussian Mixtures »
Reshad Hosseini · Suvrit Sra 
2015 Poster: On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants »
Sashank J. Reddi · Ahmed Hefny · Suvrit Sra · Barnabas Poczos · Alexander Smola 
2014 Workshop: Discrete Optimization in Machine Learning »
Jeffrey A Bilmes · Andreas Krause · Stefanie Jegelka · S Thomas McCormick · Sebastian Nowozin · Yaron Singer · Dhruv Batra · Volkan Cevher 
2014 Workshop: OPT2014: Optimization for Machine Learning »
Zaid Harchaoui · Suvrit Sra · Alekh Agarwal · Martin Jaggi · Miro Dudik · Aaditya Ramdas · Jean Lasserre · Yoshua Bengio · Amir Beck 
2014 Poster: Beyond the Birkhoff Polytope: Convex Relaxations for Vector Permutation Problems »
Cong Han Lim · Stephen Wright 
2014 Poster: Efficient Structured Matrix Rank Minimization »
Adams Wei Yu · Wanli Ma · Yaoliang Yu · Jaime Carbonell · Suvrit Sra 
2013 Workshop: OPT2013: Optimization for Machine Learning »
Suvrit Sra · Alekh Agarwal 
2013 Poster: Decision Jungles: Compact and Rich Models for Classification »
Jamie Shotton · Toby Sharp · Pushmeet Kohli · Sebastian Nowozin · John Winn · Antonio Criminisi 
2013 Poster: Geometric optimisation on positive definite matrices for elliptically contoured distributions »
Suvrit Sra · Reshad Hosseini 
2013 Poster: Reflection methods for userfriendly submodular optimization »
Stefanie Jegelka · Francis Bach · Suvrit Sra 
2013 Poster: An Approximate, Efficient LP Solver for LP Rounding »
Srikrishna Sridhar · Stephen Wright · Christopher Re · Ji Liu · Victor Bittorf · Ce Zhang 
2012 Workshop: LogLinear Models »
Dimitri Kanevsky · Tony Jebara · Li Deng · Stephen Wright · Georg Heigold · Avishy Carmi 
2012 Workshop: Optimization for Machine Learning »
Suvrit Sra · Alekh Agarwal 
2012 Poster: A new metric on the manifold of kernel matrices with application to matrix geometric means »
Suvrit Sra 
2012 Poster: Scalable nonconvex inexact proximal splitting »
Suvrit Sra 
2011 Poster: Hogwild!: A LockFree Approach to Parallelizing Stochastic Gradient Descent »
Benjamin Recht · Christopher Re · Stephen Wright · Feng Niu 
2011 Poster: HigherOrder Correlation Clustering for Image Segmentation »
Sungwoong Kim · Sebastian Nowozin · Pushmeet Kohli · Chang D. D Yoo 
2010 Workshop: Numerical Mathematics Challenges in Machine Learning »
Matthias Seeger · Suvrit Sra 
2010 Workshop: Optimization for Machine Learning »
Suvrit Sra · Sebastian Nowozin · Stephen Wright 
2010 Tutorial: Optimization Algorithms in Machine Learning »
Stephen Wright 
2009 Workshop: Optimization for Machine Learning »
Sebastian Nowozin · Suvrit Sra · S.V.N Vishwanthan · Stephen Wright 
2008 Workshop: Optimization for Machine Learning »
Suvrit Sra · Sebastian Nowozin · Vishwanathan S V N