Timezone: »
Recent work has shown that fast, compact low-bitwidth neural networks can be surprisingly accurate. These networks use homogeneous binarization: all parameters in each layer or (more commonly) the whole model have the same low bitwidth (e.g., 2 bits). However, modern hardware allows efficient designs where each arithmetic instruction can have a custom bitwidth, motivating heterogeneous binarization, where every parameter in the network may have a different bitwidth. In this paper, we show that it is feasible and useful to select bitwidths at the parameter granularity during training. For instance a heterogeneously quantized version of modern networks such as AlexNet and MobileNet, with the right mix of 1-, 2- and 3-bit parameters that average to just 1.4 bits can equal the accuracy of homogeneous 2-bit versions of these networks. Further, we provide analyses to show that the heterogeneously binarized systems yield FPGA- and ASIC-based implementations that are correspondingly more efficient in both circuit area and energy efficiency than their homogeneous counterparts.
Author Information
Joshua Fromm (University of Washington)
Shwetak Patel (University of Washington)
Matthai Philipose (Microsoft Research)
More from the Same Authors
-
2022 Poster: GLOBEM Dataset: Multi-Year Datasets for Longitudinal Human Behavior Modeling Generalization »
Xuhai Xu · Han Zhang · Yasaman Sefidgar · Yiyi Ren · Xin Liu · Woosuk Seo · Jennifer Brown · Kevin Kuehn · Mike Merrill · Paula Nurius · Shwetak Patel · Tim Althoff · Margaret Morris · Eve Riskin · Jennifer Mankoff · Anind Dey -
2021 Poster: Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection »
Chunjong Park · Anas Awadalla · Tadayoshi Kohno · Shwetak Patel -
2020 Poster: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement »
Xin Liu · Josh Fromm · Shwetak Patel · Daniel McDuff -
2020 Oral: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement »
Xin Liu · Josh Fromm · Shwetak Patel · Daniel McDuff