By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using ideas from proximal point methods in convex analysis, we prove that this BCD algorithm will converge globally to a stationary point with R-linear convergence rate of order one. In experiments with the MNIST database, DNNs trained with this BCD algorithm consistently yielded better test-set error rates than identical DNN architectures trained via all the stochastic gradient descent (SGD) variants in the Caffe toolbox.
Ziming Zhang (MERL)
Matthew Brand (Mitsubishi Electric Research Labs)
More from the Same Authors
2021 Poster: SBO-RNN: Reformulating Recurrent Neural Networks via Stochastic Bilevel Optimization »
Ziming Zhang · Yun Yue · Guojun Wu · Yanhua Li · Haichong Zhang