Timezone: »

 
Poster
FP8 Quantization: The Power of the Exponent
Andrey Kuzmin · Mart van Baalen · Yuwei Ren · Markus Nagel · Jorn Peters · Tijmen Blankevoort

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #302
in Poster Session 2 »

When quantizing neural networks for efficient inference, low-bit integers are the go-to format for efficiency. However, low-bit floating point numbers have an extra degree of freedom, assigning some bits to work on an exponential scale instead. This paper in-depth investigates this benefit of the floating point format for neural network inference. We detail the choices that can be made for the FP8 format, including the important choice of the number of bits for the mantissa and exponent, and show analytically in which settings these choices give better performance. Then we show how these findings translate to real networks, provide an efficient implementation for FP8 simulation, and a new algorithm that enables the learning of both the scale parameters and number of exponent bits in the FP8 format. Our chief conclusion is that when doing post-training quantization for a wide range of networks, the FP8 format is better than INT8 in terms of accuracy, and the choice of the number of exponent bits is driven by the severity of outliers in the network. We also conduct experiments with quantization-aware training where the difference in formats disappears as the network is trained to reduce the effect of outliers.

Keywords: post-training quantization · 8-bit floating point · quantization-aware training · Neural network quantization

Author Information

Andrey Kuzmin (Qualcomm Inc, QualComm)
Mart van Baalen (Qualcomm)
Yuwei Ren (QualComm)
Markus Nagel (Qualcomm AI Research)
Jorn Peters (University of Amsterdam)
Tijmen Blankevoort (Qualcomm)

More from the Same Authors