Timezone: »

 
Poster
Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
Roei Herzig · Moshiko Raboh · Gal Chechik · Jonathan Berant · Amir Globerson

Thu Dec 06 07:45 AM -- 09:45 AM (PST) @ Room 517 AB #140
in Thu Poster Session A »

Machine understanding of complex images is a key goal of artificial intelligence. One challenge underlying this task is that visual scenes contain multiple inter-related objects, and that global context plays an important role in interpreting the scene. A natural modeling framework for capturing such effects is structured prediction, which optimizes over complex labels, while modeling within-label interactions. However, it is unclear what principles should guide the design of a structured prediction model that utilizes the power of deep learning components. Here we propose a design principle for such architectures that follows from a natural requirement of permutation invariance. We prove a necessary and sufficient characterization for architectures that follow this invariance, and discuss its implication on model design. Finally, we show that the resulting model achieves new state of the art results on the Visual Genome scene graph labeling benchmark, outperforming all recent approaches.

Keywords: Structured Prediction · Visual Scene Analysis and Interpretation
[ Paper]  [ Poster] 

Author Information

Roei Herzig (Tel Aviv University)
Moshiko Raboh (Tel Aviv University)
Gal Chechik (NVIDIA, BIU)
Jonathan Berant (Tel Aviv University)
Amir Globerson (Tel Aviv University, Google)

Amir Globerson is senior lecturer at the School of Engineering and Computer Science at the Hebrew University. He received a PhD in computational neuroscience from the Hebrew University, and was a Rothschild postdoctoral fellow at MIT. He joined the Hebrew University in 2008. His research interests include graphical models and probabilistic inference, convex optimization, robust learning and natural language processing.

More from the Same Authors