Timezone: »
Poster
A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation
Mitsuru Kusumoto · Takuya Inoue · Gentaro Watanabe · Takuya Akiba · Masanori Koyama
Thu Dec 12 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #153
Recomputation algorithms collectively refer to a family of methods that aims to reduce the memory consumption of the backpropagation by selectively discarding the intermediate results of the forward propagation and recomputing the discarded results as needed.
In this paper, we will propose a novel and efficient recomputation method that can be applied to a wider range of neural nets than previous methods.
We use the language of graph theory to formalize the general recomputation problem of minimizing the computational overhead under a fixed memory budget constraint, and provide a dynamic programming solution to the problem.
Our method can reduce the peak memory consumption on various benchmark networks by $36\%\sim81\%$, which outperforms the reduction achieved by other methods.
Author Information
Mitsuru Kusumoto (Preferred Networks, Inc.)
Takuya Inoue (University of Tokyo)
Gentaro Watanabe (Preferred Networks, Inc.)
Takuya Akiba (Preferred Networks, Inc.)
Masanori Koyama (Preferred Networks Inc. )
More from the Same Authors
-
2021 : Contrastive Representation Learning with Trainable Augmentation Channel »
Masanori Koyama · Kentaro Minami · Takeru Miyato · Yarin Gal -
2022 Poster: Unsupervised Learning of Equivariant Structure from Sequences »
Takeru Miyato · Masanori Koyama · Kenji Fukumizu -
2019 Poster: Robustness to Adversarial Perturbations in Learning from Incomplete Data »
Amir Najafi · Shin-ichi Maeda · Masanori Koyama · Takeru Miyato