Timezone: »

 
Poster
WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models
Yonatan Bitton · Nitzan Bitton Guetta · Ron Yosef · Yuval Elovici · Mohit Bansal · Gabriel Stanovsky · Roy Schwartz

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #1020
in Poster Session 1 »

While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills. In this work, we introduce WinoGAViL: an online game of vision-and-language associations (e.g., between werewolves and a full moon), used as a dynamic evaluation benchmark. Inspired by the popular card game Codenames, a spymaster gives a textual cue related to several visual candidates, and another player tries to identify them. Human players are rewarded for creating associations that are challenging for a rival AI model but still solvable by other human players. We use the game to collect 3.5K instances, finding that they are intuitive for humans (>90% Jaccard index) but challenging for state-of-the-art AI models, where the best model (ViLT) achieves a score of 52%, succeeding mostly where the cue is visually salient. Our analysis as well as the feedback we collect from players indicate that the collected associations require diverse reasoning skills, including general knowledge, common sense, abstraction, and more. We release the dataset, the code and the interactive game, allowing future data collection that can be used to develop models with better association abilities.

Keywords: vision-and-language · visual-associations · dynamic-benchmark · visual-common-sense-reasoning
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Yonatan Bitton (The Hebrew University of Jerusalem)
Yonatan Bitton

I am a PhD candidate in The Hebrew University of Jerusalem, Israel, under the supervision of Dr. Roy Shwartz and Dr. Gabriel Stanovsky. The goal of my research is to improve vision and language generalization. Specifically, I aim to develop models with better compositionality abilities, less biased and better perform on real-world examples. See my publications for more details.

Nitzan Bitton Guetta (Ben-Gurion University)
Ron Yosef (The Hebrew University of Jerusalem)
Yuval Elovici (Ben Gurion University of the Negev, Technion)
Mohit Bansal (UNC Chapel Hill)
Gabriel Stanovsky (Hebrew University of Jerusalem)
Roy Schwartz (The Hebrew University of Jerusalem)

More from the Same Authors