Timezone: »

 
Poster
How to talk so AI will learn: Instructions, descriptions, and autonomy
Theodore Sumers · Robert Hawkins · Mark Ho · Tom Griffiths · Dylan Hadfield-Menell

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #737
in Poster Session 3 »

From the earliest years of our lives, humans use language to express our beliefs and desires. Being able to talk to artificial agents about our preferences would thus fulfill a central goal of value alignment. Yet today, we lack computational models explaining such language use. To address this challenge, we formalize learning from language in a contextual bandit setting and ask how a human might communicate preferences over behaviors. We study two distinct types of language: instructions, which provide information about the desired policy, and descriptions, which provide information about the reward function. We show that the agent's degree of autonomy determines which form of language is optimal: instructions are better in low-autonomy settings, but descriptions are better when the agent will need to act independently. We then define a pragmatic listener agent that robustly infers the speaker's reward function by reasoning about how the speaker expresses themselves. We validate our models with a behavioral experiment, demonstrating that (1) our speaker model predicts human behavior, and (2) our pragmatic listener successfully recovers humans' reward functions. Finally, we show that this form of social learning can integrate with and reduce regret in traditional reinforcement learning. We hope these insights facilitate a shift from developing agents that obey language to agents that learn from it.

Keywords: instructions · language · value alignment · descriptions · social learning · pragmatics
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Theodore Sumers (Princeton University)
Theodore Sumers

My research uses reinforcement learning and decision theory to study human communication. Theoretically, I'm interested in explaining how societies accumulate information over generations. Practically, I hope to develop artificial systems capable of interacting with and learning from humans.

Robert Hawkins (Princeton University)
Mark Ho (New York University)
Tom Griffiths (Princeton University)
Dylan Hadfield-Menell (MIT)

More from the Same Authors