Poster
The Reversible Residual Network: Backpropagation Without Storing Activations
Aidan Gomez · Mengye Ren · Raquel Urtasun · Roger Grosse
Pacific Ballroom #120
Keywords: [ Supervised Deep Networks ] [ Computer Vision ] [ Classification ] [ Efficient Training Methods ]
Residual Networks (ResNets) have demonstrated significant improvement over traditional Convolutional Neural Networks (CNNs) on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck as one needs to store all the intermediate activations for calculating gradients using backpropagation. In this work, we present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backprop. We demonstrate the effectiveness of RevNets on CIFAR and ImageNet, establishing nearly identical performance to equally-sized ResNets, with activation storage requirements independent of depth.
Live content is unavailable. Log in and register to view live content