Timezone: »

 
Poster
Associating Objects and Their Effects in Video through Coordination Games
Erika Lu · Forrester Cole · Weidi Xie · Tali Dekel · Bill Freeman · Andrew Zisserman · Michael Rubinstein

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #411
in Poster Session 4 »

We explore a feed-forward approach for decomposing a video into layers, where each layer contains an object of interest along with its associated shadows, reflections, and other visual effects. This problem is challenging since associated effects vary widely with the 3D geometry and lighting conditions in the scene, and ground-truth labels for visual effects are difficult (and in some cases impractical) to collect. We take a self-supervised approach and train a neural network to produce a foreground image and alpha matte from a rough object segmentation mask under a reconstruction and sparsity loss. Under reconstruction loss, the layer decomposition problem is underdetermined: many combinations of layers may reconstruct the input video.Inspired by the game theory concept of focal points---or \emph{Schelling points}---we pose the problem as a coordination game, where each player (network) predicts the effects for a single object without knowledge of the other players' choices. The players learn to converge on the ``natural'' layer decomposition in order to maximize the likelihood of their choices aligning with the other players'. We train the network to play this game with itself, and show how to design the rules of this game so that the focal point lies at the correct layer decomposition. We demonstrate feed-forward results on a challenging synthetic dataset, then show that pretraining on this dataset significantly reduces optimization time for real videos.

[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Erika Lu (Google)
Forrester Cole (Google Research)
Weidi Xie (University of Oxford)
Tali Dekel (Weizmann Institute of Science)
Bill Freeman (MIT/Google)
Andrew Zisserman (DeepMind & University of Oxford)
Michael Rubinstein (Google)

More from the Same Authors