Skip to yearly menu bar Skip to main content


Poster

Preventing Gradient Explosions in Gated Recurrent Units

Sekitoshi Kanai · Yasuhiro Fujiwara · Sotetsu Iwamura

Pacific Ballroom #106

Keywords: [ Time Series Analysis ] [ Recurrent Networks ] [ Supervised Deep Networks ] [ Dynamical Systems ] [ Optimization for Deep Networks ]


Abstract:

A gated recurrent unit (GRU) is a successful recurrent neural network architecture for time-series data. The GRU is typically trained using a gradient-based method, which is subject to the exploding gradient problem in which the gradient increases significantly. This problem is caused by an abrupt change in the dynamics of the GRU due to a small variation in the parameters. In this paper, we find a condition under which the dynamics of the GRU changes drastically and propose a learning method to address the exploding gradient problem. Our method constrains the dynamics of the GRU so that it does not drastically change. We evaluated our method in experiments on language modeling and polyphonic music modeling. Our experiments showed that our method can prevent the exploding gradient problem and improve modeling accuracy.

Live content is unavailable. Log in and register to view live content