Poster
Preventing Gradient Explosions in Gated Recurrent Units
Sekitoshi Kanai · Yasuhiro Fujiwara · Sotetsu Iwamura
Pacific Ballroom #106
Keywords: [ Optimization for Deep Networks ] [ Dynamical Systems ] [ Supervised Deep Networks ] [ Recurrent Networks ] [ Time Series Analysis ]
A gated recurrent unit (GRU) is a successful recurrent neural network architecture for time-series data. The GRU is typically trained using a gradient-based method, which is subject to the exploding gradient problem in which the gradient increases significantly. This problem is caused by an abrupt change in the dynamics of the GRU due to a small variation in the parameters. In this paper, we find a condition under which the dynamics of the GRU changes drastically and propose a learning method to address the exploding gradient problem. Our method constrains the dynamics of the GRU so that it does not drastically change. We evaluated our method in experiments on language modeling and polyphonic music modeling. Our experiments showed that our method can prevent the exploding gradient problem and improve modeling accuracy.
Live content is unavailable. Log in and register to view live content