Timezone: »

Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
Sainandan Ramakrishnan · Aishwarya Agrawal · Stefan Lee

Tue Dec 04 02:00 PM -- 04:00 PM (PST) @ Room 210 #79

Modern Visual Question Answering (VQA) models have been shown to rely heavily on superficial correlations between question and answer words learned during training -- \eg overwhelmingly reporting the type of room as kitchen or the sport being played as tennis, irrespective of the image. Most alarmingly, this shortcoming is often not well reflected during evaluation because the same strong priors exist in test distributions; however, a VQA system that fails to ground questions in image content would likely perform poorly in real-world settings.

In this work, we present a novel regularization scheme for VQA that reduces this effect. We introduce a question-only model that takes as input the question encoding from the VQA model and must leverage language biases in order to succeed. We then pose training as an adversarial game between the VQA model and this question-only adversary -- discouraging the VQA model from capturing language biases in its question encoding.Further, we leverage this question-only model to estimate the mutual information between the image and answer given the question, which we maximize explicitly to encourage visual grounding. Our approach is a model agnostic training procedure and simple to implement. We show empirically that it can improve performance significantly on a bias-sensitive split of the VQA dataset for multiple base models -- achieving state-of-the-art on this task. Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.

Author Information

Sainandan Ramakrishnan (Georgia Institute of Technology)

Graduate Student Researcher at GATech in the college of computing. Worked with Devi Parikh and Stefan Lee's group, now a TA for Computer Vision by James Hays.

Aishwarya Agrawal (Georgia Institute of Technology)
Stefan Lee (Georgia Institute of Technology)

More from the Same Authors