Timezone: »

 
Interactive Language: Talking to Robots in Real Time
Corey Lynch · Pete Florence · Jonathan Tompson · Ayzaan Wahid · Tianli Ding · James Betker · Robert Baruch · Travis Armstrong

@
in 5th Robot Learning Workshop: Trustworthy Robotics »

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://sites.google.com/view/interactive-language.

Author Information

Corey Lynch (Google Brain)
Pete Florence (Robotics at Google)
Jonathan Tompson (Google Brain)
Ayzaan Wahid (Google)
Tianli Ding (Google)
James Betker (Google)
Robert Baruch (Google)
Travis Armstrong (Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors