Timezone: »

 
Oral
Ultra-Low Precision 4-bit Training of Deep Neural Networks
Xiao Sun · Naigang Wang · Chia-Yu Chen · Jiamin Ni · Ankur Agrawal · Xiaodong Cui · Swagath Venkataramani · Kaoutar El Maghraoui · Vijayalakshmi (Viji) Srinivasan · Kailash Gopalakrishnan

Wed Dec 09 06:00 AM -- 06:15 AM (PST) @ Orals & Spotlights: Deep Learning
in Orals & Spotlights Track 18: Deep Learning »

In this paper, we propose a number of novel techniques and numerical representation formats that enable, for the very first time, the precision of training systems to be aggressively scaled from 8-bits to 4-bits. To enable this advance, we explore a novel adaptive Gradient Scaling technique (Gradscale) that addresses the challenges of insufficient range and resolution in quantized gradients as well as explores the impact of quantization errors observed during model training. We theoretically analyze the role of bias in gradient quantization and propose solutions that mitigate the impact of this bias on model convergence. Finally, we examine our techniques on a spectrum of deep learning models in computer vision, speech, and NLP. In combination with previously proposed solutions for 4-bit quantization of weight and activation tensors, 4-bit training shows a non-significant loss in accuracy across application domains while enabling significant hardware acceleration (> 7X over state-of-the-art FP16 systems).

Author Information

Xiao Sun (IBM Thomas J. Watson Research Center)
Naigang Wang (IBM T. J. Watson Research Center)
Chia-Yu Chen (IBM research)

my research areas focus on: accelerator architecture compiler design and library development machine learning and neural network VLSI and nano device

Jiamin Ni (IBM)
Ankur Agrawal (IBM Research)
Xiaodong Cui (IBM T. J. Watson Research Center)
Swagath Venkataramani (IBM Research)
Kaoutar El Maghraoui (IBM Research)

Dr. Kaoutar El Maghraoui is a principal research scientist at the IBM T.J Watson Research Center where she is focusing on innovations at the intersection of systems and artificial intelligence (AI). She leads the research agenda of End-Use experimental AI testbed of the IBM Research AI Hardware Center, a global research hub focusing on enabling next-generation accelerators and systems for AI workloads. . She co-led IBM’s Global Technology Outlook in 2017 where she contributed to creating IBM’s vision for the future of IT across global labs and business units focusing on IBM’s AI leadership. Kaoutar has co-authored several patents, conference, and journal publications in the areas of systems research, distributed systems, high performance computing, and AI. Kaoutar holds a PhD. degree from Rensselaer Polytechnic Institute, USA. She received several awards including the Robert McNaughton Award for best thesis in computer science, IBM’s Eminence and Excellence award for leadership in increasing Women’s presence in science and technology, and 2 IBM outstanding technical accomplishments. Kaoutar is global vice-chair of the Arab Women in Computing organization and avid supporter and volunteers of several women in science and technology initiatives.

Vijayalakshmi (Viji) Srinivasan (IBM TJ Watson)
Kailash Gopalakrishnan (IBM Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors