Skip to yearly menu bar Skip to main content


Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Tianfan Xue · Jiajun Wu · Katherine Bouman · Bill Freeman

Area 5+6+7+8 #175

Keywords: [ (Cognitive/Neuroscience) Perception ] [ (Application) Object and Pattern Recognition ] [ (Application) Computer Vision ] [ (Other) Unsupervised Learning Methods ] [ Deep Learning or Neural Networks ]


We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach which models future frames in a probabilistic manner. Our proposed method is therefore able to synthesize multiple possible next frames using the same model. Solving this challenging problem involves low- and high-level image and motion understanding for successful image synthesis. Here, we propose a novel network structure, namely a Cross Convolutional Network, that encodes images as feature maps and motion information as convolutional kernels to aid in synthesizing future frames. In experiments, our model performs well on both synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold video data. We show that our model can also be applied to tasks such as visual analogy-making, and present analysis of the learned network representations.

Live content is unavailable. Log in and register to view live content