How Many Does It Take to Prune a Network: Comparing One-Shot vs. Iterative Pruning Regimes

Pruning is one of the classical methods for network compression and model acceleration. There are two main ways pruning can be implemented: iterative pruning and one-shot pruning. In the literature, iterative pruning appears to have more proponents; however, the support for this method is often taken for granted without any substantial justification. Conversely, other approaches favor one-shot pruning. Despite the long history of pruning, there has been no thorough comparison of these two techniques. In this work, we conduct a broad and comprehensive benchmark to evaluate the regimes in both structured and unstructured pruning setting. Our findings indicate that we cannot conclusively determine which method is superior, but present scenarios where one-shot pruning is preferable for lower pruning ratios while iterative methods in higher pruning rates. This study proposes a hybrid approach that outperforms both one-shot and traditional iterative pruning.