Timezone: »

 
Spotlight
MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge
Geng Yuan · Xiaolong Ma · Wei Niu · Zhengang Li · Zhenglun Kong · Ning Liu · Yifan Gong · Zheng Zhan · Chaoyang He · Qing Jin · Siyue Wang · Minghai Qin · Bin Ren · Yanzhi Wang · Sijia Liu · Xue Lin

@

Recently, a new trend of exploring sparsity for accelerating neural network training has emerged, embracing the paradigm of training on the edge. This paper proposes a novel Memory-Economic Sparse Training (MEST) framework targeting for accurate and fast execution on edge devices. The proposed MEST framework consists of enhancements by Elastic Mutation (EM) and Soft Memory Bound (&S) that ensure superior accuracy at high sparsity ratios. Different from the existing works for sparse training, this current work reveals the importance of sparsity schemes on the performance of sparse training in terms of accuracy as well as training speed on real edge devices. On top of that, the paper proposes to employ data efficiency for further acceleration of sparse training. Our results suggest that unforgettable examples can be identified in-situ even during the dynamic exploration of sparsity masks in the sparse training process, and therefore can be removed for further training speedup on edge devices. Comparing with state-of-the-art (SOTA) works on accuracy, our MEST increases Top-1 accuracy significantly on ImageNet when using the same unstructured sparsity scheme. Systematical evaluation on accuracy, training speed, and memory footprint are conducted, where the proposed MEST framework consistently outperforms representative SOTA works. A reviewer strongly against our work based on his false assumptions and misunderstandings. On top of the previous submission, we employ data efficiency for further acceleration of sparse training. And we explore the impact of model sparsity, sparsity schemes, and sparse training algorithms on the number of removable training examples. Our codes are publicly available at: https://github.com/boone891214/MEST.

Author Information

Geng Yuan (Northeastern University)
Xiaolong Ma (Northeastern University)
Wei Niu (The College of William and Mary)
Zhengang Li (Northeastern University)
Zhenglun Kong (Northeastern University)
Ning Liu (Midea)
Yifan Gong (Northeastern University)
Zheng Zhan (Northeastern University)
Chaoyang He (University of Southern California)
Qing Jin (Northeastern University)
Siyue Wang (Google)
Minghai Qin (WDC Research)
Bin Ren (Department of Computer Science, College of William and Mary)
Yanzhi Wang (Northeastern University)
Sijia Liu (Michigan State University)
Xue Lin (Northeastern University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors