Timezone: »
Complex physical tasks entail a sequence of object interactions, each with its own preconditions -- which can be difficult for robotic agents to learn efficiently solely through their own experience. We introduce an approach to discover activity-context priors from in-the-wild egocentric video captured with human worn cameras. For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e.g., a knife and cutting board brought together with a tomato are conducive to cutting). We encode our video-based prior as an auxiliary reward function that encourages an agent to bring compatible objects together before attempting an interaction. In this way, our model translates everyday human experience into embodied agent skills. We demonstrate our idea using egocentric EPIC-Kitchens video of people performing unscripted kitchen activities to benefit virtual household robotic agents performing various complex tasks in AI2-iTHOR, significantly accelerating agent learning.
Author Information
Tushar Nagarajan (UT Austin)
Kristen Grauman (University of Texas at Austin)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: Shaping embodied agent behavior with activity-context priors from egocentric video »
Dates n/a. Room
More from the Same Authors
-
2022 Poster: SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning »
Changan Chen · Carl Schissler · Sanchit Garg · Philip Kobernik · Alexander Clegg · Paul Calamia · Dhruv Batra · Philip Robinson · Kristen Grauman -
2022 Poster: Few-Shot Audio-Visual Learning of Environment Acoustics »
Sagnik Majumder · Changan Chen · Ziad Al-Halah · Kristen Grauman -
2020 : Panel Discussion & Closing »
Yejin Choi · Alexei Efros · Chelsea Finn · Kristen Grauman · Quoc V Le · Yann LeCun · Ruslan Salakhutdinov · Eric Xing -
2020 : Q & A and Panel Session with Dan Weld, Kristen Grauman, Scott Yih, Emma Brunskill, and Alex Ratner »
Kristen Grauman · Wen-tau Yih · Alexander Ratner · Emma Brunskill · Douwe Kiela · Daniel S. Weld -
2020 : QA: Kristen Grauman »
Kristen Grauman -
2020 : Invited Talk: Kristen Grauman »
Kristen Grauman -
2020 Poster: Learning Affordance Landscapes for Interaction Exploration in 3D Environments »
Tushar Nagarajan · Kristen Grauman -
2020 Spotlight: Learning Affordance Landscapes for Interaction Exploration in 3D Environments »
Tushar Nagarajan · Kristen Grauman -
2017 Poster: Learning Spherical Convolution for Fast Features from 360° Imagery »
Yu-Chuan Su · Kristen Grauman -
2014 Poster: Diverse Sequential Subset Selection for Supervised Video Summarization »
Boqing Gong · Wei-Lun Chao · Kristen Grauman · Fei Sha -
2014 Poster: Predicting Useful Neighborhoods for Lazy Local Learning »
Aron Yu · Kristen Grauman -
2014 Poster: Zero-shot recognition with unreliable attributes »
Dinesh Jayaraman · Kristen Grauman -
2013 Poster: Reshaping Visual Datasets for Domain Adaptation »
Boqing Gong · Kristen Grauman · Fei Sha -
2012 Poster: Semantic Kernel Forests from Multiple Taxonomies »
Sung Ju Hwang · Kristen Grauman · Fei Sha -
2011 Poster: Learning a Tree of Metrics with Disjoint Visual Features »
Sung Ju Hwang · Kristen Grauman · Fei Sha -
2010 Poster: Hashing Hyperplane Queries to Near Points with Applications to Large-Scale Active Learning »
Prateek Jain · Sudheendra Vijayanarasimhan · Kristen Grauman -
2008 Oral: Multi-Level Active Prediction of Useful Image Annotations for Recognition »
Sudheendra N Vijayanarasimhan · Kristen Grauman -
2008 Poster: Multi-Level Active Prediction of Useful Image Annotations for Recognition »
Sudheendra N Vijayanarasimhan · Kristen Grauman -
2008 Poster: Online Metric Learning and Fast Similarity Search »
Prateek Jain · Brian Kulis · Inderjit Dhillon · Kristen Grauman -
2008 Oral: Online Metric Learning and Fast Similarity Search »
Prateek Jain · Brian Kulis · Inderjit Dhillon · Kristen Grauman -
2006 Poster: Approximate Correspondences in High Dimensions »
Kristen Grauman · Trevor Darrell -
2006 Spotlight: Approximate Correspondences in High Dimensions »
Kristen Grauman · Trevor Darrell