Timezone: »

Pre-training of Recurrent Neural Networks via Linear Autoencoders
Luca Pasa · Alessandro Sperduti

Mon Dec 08 04:00 PM -- 08:59 PM (PST) @ Level 2, room 210D #None

We propose a pre-training technique for recurrent neural networks based on linear autoencoder networks for sequences, i.e. linear dynamical systems modelling the target sequences. We start by giving a closed form solution for the definition of the optimal weights of a linear autoencoder given a training set of sequences. This solution, however, is computationally very demanding, so we suggest a procedure to get an approximate solution for a given number of hidden units. The weights obtained for the linear autoencoder are then used as initial weights for the input-to-hidden connections of a recurrent neural network, which is then trained on the desired task. Using four well known datasets of sequences of polyphonic music, we show that the proposed pre-training approach is highly effective, since it allows to largely improve the state of the art results on all the considered datasets.

Author Information

Luca Pasa (Università degli Studi di Padova)
Alessandro Sperduti (Università degli Studi di Padova)