Timezone: »

 
Poster
PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs
Yunbo Wang · Mingsheng Long · Jianmin Wang · Zhifeng Gao · Philip S Yu

Mon Dec 04 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #115
in Posters Mon »

The predictive learning of spatiotemporal sequences aims to generate future images by learning from the historical frames, where spatial appearances and temporal variations are two crucial structures. This paper models these structures by presenting a predictive recurrent neural network (PredRNN). This architecture is enlightened by the idea that spatiotemporal predictive learning should memorize both spatial appearances and temporal variations in a unified memory pool. Concretely, memory states are no longer constrained inside each LSTM unit. Instead, they are allowed to zigzag in two directions: across stacked RNN layers vertically and through all RNN states horizontally. The core of this network is a new Spatiotemporal LSTM (ST-LSTM) unit that extracts and memorizes spatial and temporal representations simultaneously. PredRNN achieves the state-of-the-art prediction performance on three video prediction datasets and is a more general framework, that can be easily extended to other predictive learning tasks by integrating with other architectures.

Keywords: Recurrent Networks · Predictive Models
[ Code]  [ Paper] 

Author Information

Yunbo Wang (Tsinghua University)
Mingsheng Long (Tsinghua University)
Jianmin Wang (Tsinghua University)
Zhifeng Gao (Tsinghua University)
Philip S Yu (UIC)

More from the Same Authors