Timezone: »

A Cooperative Visually Grounded Dialogue Game with a Humanoid Robot
Jordan Prince Tremblay · Ismael Balafrej · Felix Labelle · Félix Martel-Denis · Eric Matte · Julien Chouinard-Beaupré · Adam Letourneau · Antoine Mercier-Nicol · Simon Brodeur · François Ferland · Jean ROUAT

Wed Dec 05 07:45 AM -- 04:30 PM (PST) @ Room 510 ABCD #D9

This demonstration illustrates how research results in grounded language learning and understanding can be used in a cooperative task between an intelligent agent and a human. The task, undertaken by a robot, is the question answering game GuessWhat?! [1][2]

Providing human-robot interactions in the real world requires interfacing GuessWhat?! with: speech recognition and synthesis modules; video processing and recognition algorithms; the robot’s control module. One main challenge is adapting GuessWhat?! to work with images outside of MSCOCO’s domain. This required implementing a pipeline in ROS which takes images from a Kinect, ensures image quality with blur detection, extracts VGG-16 feature vectors, segments objects using Mask R-CNN, and extracts position information from the segmented objects. Images from the pipeline are used by GuessWhat?! in tandem with utterances from the player. Snips voice assistant recognizes whether the player says “Yes”, “No” or “Not Applicable". Snips also provides speech synthesis, converting questions generated by GuessWhat?! into speech for the player. To identify potential players, OpenPose allows IRL-1 to interact with them throughout the game.

Our open source code could be useful as intelligent agents are becoming commonplace and the ability to communicate with people in a given context, such as the home or workplace, becomes imperative. The various functionalities implemented on IRL-1 [3], would be beneficial to any agent assisting a person in a cooperative task.

More details can be found at: https://devine.gel.usherbrooke.ca/abstract_devine.pdf

[1] https://www.guesswhat.ai

[2] https://iglu-chistera.github.io

[3] http://humanrobotinteraction.org/journal/index.php/HRI/article/view/65

Author Information

Jordan Prince Tremblay (Université de Sherbooke)
Ismael Balafrej (Université de Sherbrooke)

Hi, I'm Ismael and I'm an undergraduate student in computer engineering.

Felix Labelle (University of Sherbrooke)
Félix Martel-Denis (Université de Sherbrooke)
Eric Matte (Université de Sherbrooke)
Julien Chouinard-Beaupré (Université Sherbrooke)
Adam Letourneau (University of Sherbrooke)
Antoine Mercier-Nicol (Université de Sherbrooke)
Simon Brodeur (Université de Sherbrooke)
François Ferland (Université de Sherbrooke)
Jean ROUAT (Université de Sherbrooke)

More from the Same Authors

  • 2018 : Poster Sessions and Lunch (Provided) »
    Akira Utsumi · Alane Suhr · Ji Zhang · Ramon Sanabria · Kushal Kafle · Nicholas Chen · Seung Wook Kim · Aishwarya Agrawal · SRI HARSHA DUMPALA · Shikhar Murty · Pablo Azagra · Jean ROUAT · Alaaeldin Ali · · SUBBAREDDY OOTA · Angela Lin · Shruti Palaskar · Farley Lai · Amir Aly · Tingke Shen · Dianqi Li · Jianguo Zhang · Rita Kuznetsova · Jinwon An · Jean-Benoit Delbrouck · Tomasz Kornuta · Syed Ashar Javed · Christopher Davis · John Co-Reyes · Vasu Sharma · Sungwon Lyu · Ning Xie · Ankita Kalra · Huan Ling · Oleksandr Maksymets · Bhavana Mahendra Jain · Shun-Po Chuang · Sanyam Agarwal · Jerome Abdelnour · Yufei Feng · vincent albouy · Siddharth Karamcheti · Derek Doran · Roberta Raileanu · Jonathan Heek
  • 2017 : Break + Poster (1) »
    Devendra Singh Chaplot · CHIH-YAO MA · Simon Brodeur · Eri Matsuo · Ichiro Kobayashi · Seitaro Shinagawa · Koichiro Yoshino · Yuhong Guo · Ben Murdoch · Kanthashree Mysore Sathyendra · Daniel Ricks · Haichao Zhang · Joshua Peterson · Li Zhang · Mircea Mironenco · Peter Anderson · Mark Johnson · Kang Min Yoo · Guntis Barzdins · Ahmed H Zaidi · Martin Andrews · Sam Witteveen · SUBBAREDDY OOTA · Prashanth Vijayaraghavan · Ke Wang · Yan Zhu · Renars Liepins · Max Quinn · Amit Raj · Vincent Cartillier · Eric Chu · Ethan Caballero · Fritz Obermeyer
  • 2006 Demonstration: RN-Spikes: A bio-inspired neural processor for face recognition »
    Jocely Bergeron · Stephane Loiselle · Hoang Hai Anh · Jean Lavoie · Ramin Pichevar · Vincent Lapointe · Jean ROUAT · Jean Bélanger