Skip to yearly menu bar Skip to main content


Poster
in
Workshop: LaReL: Language and Reinforcement Learning

On the Pitfalls of Visual Learning in Referential Games

Shresth Verma

Keywords: [ Reinforcement Learning ] [ language emergence ] [ referential games ]


Abstract:

This paper focuses on the effect of game design and visual representations of real-world entities on emergent languages in referential games. Strikingly, we find that the agents in such games can learn to successfully communicate even when provided with visual features from a randomly initialized neural network. Through a series of experiments, we highlight the agents' inability to effectively utilize high-level features. Using Gradient weighted-Class Activation Mapping, we verify that the agents often 'look' at regions not related to entities.Culminating with a positive result, we show how environmental pressure from agent population can nudge the learners into effectively capturing high-level visual features.

Chat is not available.