Timezone: »

 
Poster
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
Urs Köster · Tristan Webb · Xin Wang · Marcel Nassar · Arjun K Bansal · William Constable · Oguz Elibol · Stewart Hall · Luke Hornof · Amir Khosrowshahi · Carey Kloss · Ruby J Pai · Naveen Rao

Mon Dec 04 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #75
in Posters Mon »

Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the Flexpoint data format, aiming at a complete replacement of 32-bit floating point format training and inference, designed to support modern deep network topologies without modifications. Flexpoint tensors have a shared exponent that is dynamically adjusted to minimize overflows and maximize available dynamic range. We validate Flexpoint by training AlexNet, a deep residual network and a generative adversarial network, using a simulator implemented with the \emph{neon} deep learning framework. We demonstrate that 16-bit Flexpoint closely matches 32-bit floating point in training all three models, without any need for tuning of model hyperparameters. Our results suggest Flexpoint as a promising numerical format for future hardware for training and inference.

Keywords: Deep Learning · Efficient Inference Methods · Efficient Training Methods · Hardware and Systems
[ Paper] 

Author Information

Urs Köster (Intel Corporation)
Tristan Webb (Intel / Nervana)
Xin Wang (Intel Corporation)

https://www.linkedin.com/in/neuromorphic

Marcel Nassar (Intel Corporation)
Arjun K Bansal (Intel Nervana)
William Constable (Intel)
Oguz Elibol (Intel Nervana)
Stewart Hall (Intel)
Luke Hornof (Intel Nervana)
Amir Khosrowshahi (Intel AI)
Carey Kloss (Intel)
Ruby J Pai (Intel Corporation)
Naveen Rao (Intel)

Trained as both a computer architect and neuroscientist, Rao joined Intel in 2016 with the acquisition of Nervana Systems. As chief executive officer and co-founder of Nervana, he led the company to become a recognized leader in the deep learning field. Before founding Nervana in 2014, Rao was a neuromorphic machines researcher at Qualcomm Inc., where he focused on neural computation and learning in artificial systems. Rao’s earlier career included engineering roles at Kealia Inc., CALY Networks and Sun Microsystems Inc. Rao earned a bachelor’s degree in electrical engineering and computer science from Duke University, then spent a decade as a computer architect before going on to earn a Ph.D. in computational neuroscience from Brown University. He has published multiple papers in the area of neural computation in biological systems. Rao has also been granted patents in video compression techniques, with additional patents pending in deep learning hardware and low-precision techniques and in neuromorphic computation.

More from the Same Authors