Timezone: »

 
Poster
Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence
Fengxiang He · Tongliang Liu · Dacheng Tao

Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #219
in Theory -- Learning Theory »
Deep neural networks have received dramatic success based on the optimization method of stochastic gradient descent (SGD). However, it is still not clear how to tune hyper-parameters, especially batch size and learning rate, to ensure good generalization. This paper reports both theoretical and empirical evidence of a training strategy that we should control the ratio of batch size to learning rate not too large to achieve a good generalization ability. Specifically, we prove a PAC-Bayes generalization bound for neural networks trained by SGD, which has a positive correlation with the ratio of batch size to learning rate. This correlation builds the theoretical foundation of the training strategy. Furthermore, we conduct a large-scale experiment to verify the correlation and training strategy. We trained 1,600 models based on architectures ResNet-110, and VGG-19 with datasets CIFAR-10 and CIFAR-100 while strictly control unrelated variables. Accuracies on the test sets are collected for the evaluation. Spearman's rank-order correlation coefficients and the corresponding $p$ values on 164 groups of the collected data demonstrate that the correlation is statistically significant, which fully supports the training strategy.
Keywords: Deep Learning · Optimization for Deep Networks · Deep Learning
[ Paper]  [ Poster] 

Author Information

Fengxiang He (The University of Sydney)
Tongliang Liu (The University of Sydney)
Dacheng Tao (University of Sydney)

More from the Same Authors