Timezone: »

Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network
Lixin Fan

Mon Dec 04 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #227 #None

We revisit fuzzy neural network with a cornerstone notion of generalized hamming distance, which provides a novel and theoretically justified framework to re-interpret many useful neural network techniques in terms of fuzzy logic. In particular, we conjecture and empirically illustrate that, the celebrated batch normalization (BN) technique actually adapts the “normalized” bias such that it approximates the rightful bias induced by the generalized hamming distance. Once the due bias is enforced analytically, neither the optimization of bias terms nor the sophisticated batch normalization is needed. Also in the light of generalized hamming distance, the popular rectified linear units (ReLU) can be treated as setting a minimal hamming distance threshold between network inputs and weights. This thresholding scheme, on the one hand, can be improved by introducing double-thresholding on both positive and negative extremes of neuron outputs. On the other hand, ReLUs turn out to be non-essential and can be removed from networks trained for simple tasks like MNIST classification. The proposed generalized hamming network (GHN) as such not only lends itself to rigorous analysis and interpretation within the fuzzy logic theory but also demonstrates fast learning speed, well-controlled behaviour and state-of-the-art performances on a variety of learning tasks.

Author Information

Lixin Fan (Nokia Technologies)

Dr Lixin Fan is a principal scientist at Nokia Technologies. His research areas of interests include Image and video processing, Computer vision, Machine learning, Map 3D data processing and rendering, Intelligent human-computer interface, Augmented and virtual reality, Mobile ubiquitous and pervasive computing. Dr Fan is the (co-)author of more than 50 international journal & conference publications. He also (co-)invented dozens of granted and pending patents filed in US, Europe and China. Before joining Nokia in 2004, Dr Fan was affiliated with Xerox Research Center Europe and his research work included the well recognized Bag of Keypoints method for image categorization. Dr Fan received his MSc and PhD in Computer Science from National University of Singapore in 1998 and 2002 respectively.