Timezone: »

 
A Mixture-of-Expert Approach to RL-based Dialogue Management
Yinlam Chow · Azamat Tulepbergenov · Ofir Nachum · Dhawal Gupta · Moonkyung Ryu · Mohammad Ghavamzadeh · Craig Boutilier

@
in Foundation Models for Decision Making »
Event URL: https://openreview.net/forum?id=b64E-oAvIM »

Despite recent advancements in language models (LMs), their application to dialogue management (DM) problems and ability to carry on rich conversations remain a challenge. We use reinforcement learning (RL) to develop a dialogue agent that avoids being short-sighted (outputting generic utterances) and maximizes overall user satisfaction. Most existing RL approaches to DM train the agent at the word-level, and thus, have to deal with a combinatorially complex action space even for a medium-size vocabulary. As a result, they struggle to produce a successful and engaging dialogue even if they are warm-started with a pre-trained LM. To address this issue, we develop a RL-based DM using a novel mixture of expert language model (MoE-LM) that consists of (i) a LM capable of learning diverse semantics for conversation histories, (ii) a number of specialized LMs (or experts) capable of generating utterances corresponding to a particular attribute or personality, and (iii) a RL-based DM that performs dialogue planning with the utterances generated by the experts. Our MoE approach provides greater flexibility to generate sensible utterances with different intents and allows RL to focus on conversational-level DM. We compare it with SOTA baselines on open-domain dialogues and demonstrate its effectiveness both in terms of the diversity and sensibility of the generated utterances and the overall DM performance.

Author Information

Yinlam Chow (Google Research)
Azamat Tulepbergenov (Google)

I am a Research Software Engineer at Google AI. I am interested in applying Reinforcement Learning to help solve useful problems. My current research interests are language, health and urban engineering.

Ofir Nachum (Google Brain)
Dhawal Gupta (University of Massachusetts Amherst)
Moonkyung Ryu (Google)
Mohammad Ghavamzadeh (Google Research)
Craig Boutilier (Google)

More from the Same Authors