This is the public, feature-limited version of the conference webpage. After Registration and login please visit the full version.

Workshop: Human in the loop dialogue systems

Behnam Hedayatnia, Rahul Goel, Shereen Oraby, Abigail See, Chandra Khatri, Y-Lan Boureau, Alborz Geramifard, Marilyn Walker, Dilek Hakkani-Tur

2020-12-11T06:10:00-08:00 - 2020-12-11T17:20:00-08:00
Abstract: Conversational interaction systems such as Amazon Alexa, Google Assistant, Apple Siri, and Microsoft Cortana have become very popular over the recent years. Such systems have allowed users to interact with a wide variety of content on the web through a conversational interface. Research challenges such as the Dialogue System Technology Challenges, Dialogue Dodecathlon, Amazon Alexa Prize and the Vision and Language Navigation task have continued to inspire research in conversational AI. These challenges have brought together researchers from different communities such as speech recognition, spoken language understanding, reinforcement learning, language generation, and multi-modal question answering.
Unlike other popular NLP tasks, dialogue frequently has humans in the loop, whether it is for evaluation, active learning or online reward estimation. Through this workshop we aim to bring together researchers from academia and industry to discuss the challenges and opportunities in such human in the loop setups. We hope that this sparks interesting discussions about conversational agents, interactive systems, and how we can use humans most effectively when building such setups. We will highlight areas such as human evaluation setups, reliability in human evaluation, human in the loop training, interactive learning and user modeling. We also highly encourage non-English based dialogue systems in these areas.
The one-day workshop will include talks from senior technical leaders and researchers to share insights associated with evaluating dialogue systems. We also plan on having oral presentations and poster sessions on works related to the topic of the workshop. Finally we will end the workshop with an interactive panel of speakers. As an outcome we expect the participants from the NeurIPS community to walk away with better understanding of human in the loop dialogue modeling as well as key areas of research in this field. Additionally we would like to see discussions around the unification of human evaluation setups in some way.

Chat

To ask questions please use rocketchat, available only upon registration and login.

Schedule

2020-12-11T06:10:00-08:00 - 2020-12-11T06:20:00-08:00
Welcome and Opening Remarks
Behnam Hedayatnia
2020-12-11T07:50:00-08:00 - 2020-12-11T08:05:00-08:00
Invited Talk 1 Q/A - Milica Gašić
Milica Gasic
2020-12-11T08:05:00-08:00 - 2020-12-11T08:20:00-08:00
Invited Talk 2 Q/A - Larry Heck
Larry Heck
2020-12-11T08:20:00-08:00 - 2020-12-11T08:35:00-08:00
Invited Talk 3 Q/A - Maxine Eskenazi
Maxine Eskenazi, Shikib Mehri
2020-12-11T09:05:00-08:00 - 2020-12-11T09:15:00-08:00
Contributed Talk 1 Q/A
Aaron Jhan
2020-12-11T09:15:00-08:00 - 2020-12-11T09:25:00-08:00
Contributed Talk 2 Q/A
Thibault Cordier
2020-12-11T09:25:00-08:00 - 2020-12-11T10:30:00-08:00
Poster Session Presentations
Nalin Chhibber, Weiyi Lu, Lina Rojas-Barahona, Katie Stasaski, Dookun Park, Govind Thattai, Alexandry Augustin, Mathilde Veron, Sahisnu Mazumder, Evgeny Krivosheev, Alessandro Bozzon
2020-12-11T10:30:00-08:00 - 2020-12-11T11:00:00-08:00
Breakout session: Human Evaluation
Behnam Hedayatnia
2020-12-11T10:30:00-08:00 - 2020-12-11T11:00:00-08:00
Breakout session: Automatic Evaluation
Yang Liu
2020-12-11T11:20:00-08:00 - 2020-12-11T11:30:00-08:00
Contributed Talk 3 Q/A
José David Águas Lopes
2020-12-11T11:30:00-08:00 - 2020-12-11T11:40:00-08:00
Contributed Talk 4 Q/A
Qiuyuan Huang, Kezhen Chen
2020-12-11T12:50:00-08:00 - 2020-12-11T13:05:00-08:00
Invited Talk 4 Q/A - Jason Weston
Jason E Weston
2020-12-11T13:05:00-08:00 - 2020-12-11T13:20:00-08:00
Invited Talk 5 Q/A - Zhou Yu
Zhou Yu
2020-12-11T13:20:00-08:00 - 2020-12-11T13:35:00-08:00
Invited Talk 6 Q/A - Gokhan Tür
Gokhan Tur
2020-12-11T13:55:00-08:00 - 2020-12-11T14:05:00-08:00
Contributed Talk 5 Q/A
Nathan Ng
2020-12-11T14:05:00-08:00 - 2020-12-11T14:20:00-08:00
Contributed Talk 6 Q/A
Chien-Wei Lin
2020-12-11T15:30:00-08:00 - 2020-12-11T15:45:00-08:00
Invited Talk 7 Q/A - Ankur Parikh
Ankur Parikh
2020-12-11T15:45:00-08:00 - 2020-12-11T16:00:00-08:00
Invited Talk 8 Q/A - Percy Liang
Percy Liang
2020-12-11T16:00:00-08:00 - 2020-12-11T16:15:00-08:00
Invited Talk 9 Q/A - Alexander Rudnicky
Alex Rudnicky
2020-12-11T16:15:00-08:00 - 2020-12-11T17:15:00-08:00
Panel
Maxine Eskenazi, Larry Heck, Ankur Parikh, Govind Thattai, Alex Rudnicky, Jason E Weston
2020-12-11T17:15:00-08:00 - 2020-12-11T17:20:00-08:00
Closing Remarks / Best Paper Award
Behnam Hedayatnia
Invited Talk 1 Presentation - Milica Gašić
Milica Gasic
Invited Talk 9 Presentation - Alexander Rudnicky
Alex Rudnicky
Invited Talk 6 Presentation - Gokhan Tür
Gokhan Tur
Invited Talk 2 Presentation - Larry Heck
Larry Heck
Invited Talk 4 Presentation - Jason Weston
Jason E Weston
Invited Talk 7 Presentation - Ankur Parikh
Ankur Parikh
Invited Talk 3 Presenation - Maxine Eskenazi
Maxine Eskenazi
Contributed Talk 5 Presentation
Nathan Ng
Invited Talk 5 Presentation - Zhou Yu
Zhou Yu
Contributed Talk 2 Presentation
Thibault Cordier
Contributed Talk 3 Presentation
José David Águas Lopes
Contributed Talk 4 Presentation
Qiuyuan Huang
Contributed Talk 6 Presentation
Chien-Wei Lin
Invited Talk 8 Presentation - Percy Liang
Percy Liang
Contributed Talk 1 Presentation
Aaron Jhan