Winning Tickets from Random Initialization: Aligning Masks for Sparse Training

The Lottery Ticket Hypothesis (LTH) suggests that there exists a sparse $\textit{winning ticket}$ mask and weights that achieves the same generalization performance as the dense model while using much fewer parameters. LTH achieves this by iteratively sparsifying and re-training within the pruned solution basin. This procedure is expensive, and any other random initialization sparsified using the winning ticket mask fails to achieve good generalization performance. Recent work has suggested that Deep Neural Networks (DNNs) trained from random initialization find solutions within the same basin modulo weight symmetry, and proposes a method to align trained models within the same basins. We propose permuting the winning ticket mask to align with the new optimization basin when performing sparse trainingfrom a different random initialization than the one used to derive the pruned mask. Using this permuted mask, we show it is possible to significantly increase the generalization performance of sparse training from random initialization as compared to sparse training naively using the non-permuted mask