Timezone: »

 
Spotlight
On Power Laws in Deep Ensembles
Ekaterina Lobacheva · Nadezhda Chirkova · Maxim Kodryan · Dmitry Vetrov

Thu Dec 10 07:10 AM -- 07:20 AM (PST) @ Orals & Spotlights: Deep Learning
in Orals & Spotlights Track 28: Deep Learning »

Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement. In this work, we focus on a classification problem and investigate the behavior of both non-calibrated and calibrated negative log-likelihood (CNLL) of a deep ensemble as a function of the ensemble size and the member network size. We indicate the conditions under which CNLL follows a power law w. r. t. ensemble size or member network size, and analyze the dynamics of the parameters of the discovered power laws. Our important practical finding is that one large network may perform worse than an ensemble of several medium-size networks with the same total number of parameters (we call this ensemble a memory split). Using the detected power law-like dependencies, we can predict (1) the possible gain from the ensembling of networks with given structure, (2) the optimal memory split given a memory budget, based on a relatively small number of trained networks.

Author Information

Ekaterina Lobacheva (Samsung-HSE Laboratory)
Nadezhda Chirkova (Higher School of Economics, Samsung-HSE Laboratory)
Maxim Kodryan (Samsung-HSE Laboratory, National Research University Higher School of Economics)
Dmitry Vetrov (Higher School of Economics, Samsung AI Center, Moscow)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors