Timezone: »
A deeper network structure generally handles more complicated non-linearity and performs more competitively. Nowadays, advanced network designs often contain a large number of repetitive structures (e.g., Transformer). They empower the network capacity to a new level but also increase the model size inevitably, which is unfriendly to either model restoring or transferring. In this study, we are the first to investigate the representative potential of fixed random weights with limited unique values by learning diverse masks and introduce the Parameter-Efficient Masking Networks (PEMN). It also naturally leads to a new paradigm for model compression to diminish the model size. Concretely, motivated by the repetitive structures in modern neural networks, we utilize one random initialized layer, accompanied with different masks, to convey different feature mappings and represent repetitive network modules. Therefore, the model can be expressed as \textit{one-layer} with a bunch of masks, which significantly reduce the model storage cost. Furthermore, we enhance our strategy by learning masks for a model filled by padding a given random weights vector. In this way, our method can further lower the space complexity, especially for models without many repetitive architectures. We validate the potential of PEMN learning masks on random weights with limited unique values and test its effectiveness for a new compression paradigm based on different network architectures.Code is available at \href{https://github.com/yueb17/PEMN}{\textcolor{magenta}{https://github.com/yueb17/PEMN}}.
Author Information
Yue Bai (Northeastern University)
Huan Wang
Xu Ma (Northeastern University)
Yitian Zhang (Northeastern University)
Zhiqiang Tao (Santa Clara University)
Yun Fu (Northeastern University)
More from the Same Authors
-
2021 Spotlight: Aligned Structured Sparsity Learning for Efficient Image Super-Resolution »
Yulun Zhang · Huan Wang · Can Qin · Yun Fu -
2022 Poster: Look More but Care Less in Video Recognition »
Yitian Zhang · Yue Bai · Huan Wang · Yi Xu · Yun Fu -
2022 Poster: What Makes a "Good" Data Augmentation in Knowledge Distillation - A Statistical Perspective »
Huan Wang · Suhas Lohit · Michael Jones · Yun Fu -
2021 Poster: Slow Learning and Fast Inference: Efficient Graph Similarity Computation via Knowledge Distillation »
Can Qin · Handong Zhao · Lichen Wang · Huan Wang · Yulun Zhang · Yun Fu -
2021 Poster: Aligned Structured Sparsity Learning for Efficient Image Super-Resolution »
Yulun Zhang · Huan Wang · Can Qin · Yun Fu -
2020 Poster: Learning to Mutate with Hypergradient Guided Population »
Zhiqiang Tao · Yaliang Li · Bolin Ding · Ce Zhang · Jingren Zhou · Yun Fu -
2020 Poster: Neural Sparse Representation for Image Restoration »
Yuchen Fan · Jiahui Yu · Yiqun Mei · Yulun Zhang · Yun Fu · Ding Liu · Thomas Huang -
2019 Poster: PointDAN: A Multi-Scale 3D Domain Adaption Network for Point Cloud Representation »
Can Qin · Haoxuan You · Lichen Wang · C.-C. Jay Kuo · Yun Fu -
2017 Poster: Matching on Balanced Nonlinear Representations for Treatment Effects Estimation »
Sheng Li · Yun Fu -
2012 Poster: Fast Resampling Weighted v-Statistics »
Chunxiao Zhou · jiseong Park · Yun Fu -
2012 Spotlight: Fast Resampling Weighted v-Statistics »
Chunxiao Zhou · jiseong Park · Yun Fu