Timezone: »
Recent advances in deep reinforcement learning and robotics have enabled agents to achieve superhuman performance on a variety of challenging games and learn complex manipulation tasks. While these results are very promising, several open problems remain. In order to function in real-world environments, learned policies must be both robust to input perturbations and be able to rapidly generalize or adapt to novel situations. Moreover, to collaborate and live with humans in these environments, the goals and actions of embodied agents must be interpretable and compatible with human representations of knowledge. Hence, it is natural to consider how humans so successfully perceive, learn, and plan to build agents that are equally successful at solving real world tasks.
There is much evidence to suggest that objects are a core level of abstraction at which humans perceive and understand the world [8]. Objects have the potential to provide a compact, casual, robust, and generalizable representation of the world. Recently, there have been many advancements in scene representation, allowing scenes to be represented by their constituent objects, rather than at the level of pixels. While these works have shown promising results, there is still a lack of agreement on how to best represent objects, how to learn object representations, and how best to leverage them in agent training.
In this workshop we seek to build a consensus on what object representations should be by engaging with researchers from developmental psychology and by defining concrete tasks and capabilities that agents building on top of such abstract representations of the world should succeed at. We will discuss how object representations may be learned through invited presenters with expertise both in unsupervised and supervised object representation learning methods. Finally, we will host conversations and research on new frontiers in object learning.
Fri 8:00 a.m. - 8:15 a.m.
|
Introduction
|
William Agnew 🔗 |
Fri 8:15 a.m. - 9:00 a.m.
|
Keynote: Elizabeth Spelke
(
Talk
)
Elizabeth Spelke is the Marshall L. Berkman Professor of Psychology at Harvard University and an investigator at the NSF-MIT Center for Brains, Minds and Machines. Her laboratory focuses on the sources of uniquely human cognitive capacities, including capacities for formal mathematics, for constructing and using symbols, and for developing comprehensive taxonomies of objects. She probes the sources of these capacities primarily through behavioral research on human infants and preschool children, focusing on the origins and development of their understanding of objects, actions, people, places, number, and geometry. In collaboration with computational cognitive scientists, she aims to test computational models of infants’ cognitive capacities. In collaboration with economists, she has begun to take her research from the laboratory to the field, where randomized controlled experiments can serve to evaluate interventions, guided by research in cognitive science, that seek to enhance young children’s learning. |
🔗 |
Fri 9:02 a.m. - 9:04 a.m.
|
Learning Object-Centric Video Models by Contrasting Sets
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:04 a.m. - 9:06 a.m.
|
Structure-Regularized Attention for Deformable Object Representation
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:06 a.m. - 9:08 a.m.
|
Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:08 a.m. - 9:10 a.m.
|
Self-Supervised Attention-Aware Reinforcement Learning
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:10 a.m. - 9:12 a.m.
|
Emergence of compositional abstractions in human collaborative assembly
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:12 a.m. - 9:14 a.m.
|
Semantic State Representation for Reinforcement Learning
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:14 a.m. - 9:16 a.m.
|
Odd-One-Out Representation Learning
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:16 a.m. - 9:18 a.m.
|
Word(s) and Object(s): Grounded Language Learning In Information Retrieval
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:20 a.m. - 9:22 a.m.
|
Discrete Predictive Representation for Long-horizon Planning
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:22 a.m. - 9:24 a.m.
|
Dynamic Regions Graph Neural Networks for Spatio-Temporal Reasoning
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:26 a.m. - 9:28 a.m.
|
Dexterous Robotic Grasping with Object-Centric Visual Affordances
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:28 a.m. - 9:30 a.m.
|
Understanding designed objects by program synthesis
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:29 a.m. - 9:31 a.m.
|
Learning Embeddings that Capture Spatial Semantics for Indoor Navigation
(
Lightning
)
SlidesLive Video » |
🔗 |
Fri 9:30 a.m. - 10:30 a.m.
|
Poster Session A in GatherTown ( Poster Session ) link » | 🔗 |
Fri 10:30 a.m. - 11:45 a.m.
|
Panel Discussion
How can we obtain object representations in real world environments? How can object representations be applied in robotics? Join us for a panel discussion with Jessica Hamrick, Irina Higgins, Michelle Lee, Josh Tenenbaum, moderated by Klaus Greff. |
Jessica Hamrick · Klaus Greff · Michelle A. Lee · Irina Higgins · Josh Tenenbaum 🔗 |
Fri 11:45 a.m. - 12:25 p.m.
|
Break in GatherTown link » | 🔗 |
Fri 12:25 p.m. - 12:55 p.m.
|
Invited Talk: Jessica Hamrick
(
Talk
)
SlidesLive Video » Jessica Hamrick is a Senior Research Scientist at DeepMind, where she studies how to build machines that can flexibly build and deploy models of the world. Her work combines insights from cognitive science with structured relational architectures, model-based deep reinforcement learning, and planning. Jessica received a Ph.D. in Psychology from UC Berkeley in 2017, and an M.Eng. and B.S. in Computer Science from MIT in 2012. |
Jessica Hamrick 🔗 |
Fri 12:55 p.m. - 1:25 p.m.
|
Invited Talk: Irina Higgins
(
Talk
)
SlidesLive Video » Irina Higgins is a research scientist at DeepMind, where she works in the Frontiers team. Her work aims to bring together insights from the fields of neuroscience and physics to advance general artificial intelligence through improved representation learning. Before joining DeepMind, Irina was a British Psychological Society Undergraduate Award winner for her achievements as an undergraduate student in Experimental Psychology at Westminster University, followed by a DPhil at the Oxford Centre for Computational Neuroscience and Artificial Intelligence, where she focused on understanding the computational principles underlying speech processing in the auditory brain. During her DPhil, Irina also worked on developing poker AI, applying machine learning in the finance sector, and working on speech recognition at Google Research. |
Irina Higgins 🔗 |
Fri 1:25 p.m. - 1:55 p.m.
|
Invited Talk: Sungjin Ahn
(
Talk
)
SlidesLive Video » Sungjin Ahn is an Assistant Professor of Computer Science at Rutgers University and directs the Rutgers Machine Learning (RUML) lab. He is also affiliated with Rutgers Center for Cognitive Science. His research focus is on how an AI-agent can learn the structure and representations of the world in an unsupervised and compositional way, with a particular interest in object-centric learning. His approach to achieving this is based on deep learning, Bayesian modeling, reinforcement learning, and inspiration from cognitive & neuroscience. He received Ph.D. at the University of California, Irvine with Max Welling and did a postdoc with Yoshua Bengio at Mila. Then, he joined Rutgers University in Fall 2018. He has co-organized ICML 2020 Workshop on Object-Oriented Learning and received the ICML best paper award in ICML 2012. |
Sungjin Ahn 🔗 |
Fri 1:55 p.m. - 2:07 p.m.
|
Contributed Talk : A Symmetric and Object-Centric World Model for Stochastic Environments
(
Talk
)
SlidesLive Video » |
🔗 |
Fri 2:07 p.m. - 2:19 p.m.
|
Contributed Talk : OGRE: An Object-based Generalization for Reasoning Environment
(
Talk
)
SlidesLive Video » |
🔗 |
Fri 2:19 p.m. - 2:49 p.m.
|
Invited Talk: Wilka Carvalho
(
Talk
)
SlidesLive Video » Wilka Carvalho is a PhD Candidate in Computer Science at the University of Michigan–Ann Arbor where he is advised by Honglak Lee, Satinder Singh, and Richard Lewis. His long-term research goal is to develop cognitive theories of learning that help us understand how humans infer, reason with, and exploit the rich structure present in realistic visual scenes to enable sophisticated behavioral policies. Towards this end, he is studying how object-centric representation learning and reinforcement learning can bring us closer to human-level artificial intelligence. He is supported by an NSF GRFP Fellowship and a UM Rackham Merit Fellowship. |
Wilka Carvalho 🔗 |
Fri 2:49 p.m. - 3:20 p.m.
|
Break in GatherTown link » | 🔗 |
Fri 3:20 p.m. - 3:50 p.m.
|
Invited Talk: Renée Baillargeon
(
Talk
)
SlidesLive Video » Renée Baillargeon is an Alumni Distinguished Professor of Psychology at the University of Illinois Urbana-Champaign. Her research examines cognitive development in infancy and focuses primarily on causal reasoning. In particular, she explores how infants make sense of the events they observe, and what explanatory frameworks and learning mechanisms enable them to do so. In addition to this primary focus on causal reasoning, she is interested in a broad range of related issues including object perception, categorization, object individuation, number, and executive-function skills. |
🔗 |
Fri 3:50 p.m. - 4:20 p.m.
|
Invited Talk: Dieter Fox
(
Talk
)
Object Representations for Robot Manipulation Reasoning about objects is a fundamental task in robot manipulation. Different representations can have important repercussions on the capabilities and generality of a manipulation system. In this talk I will discuss different ways we represent and reason about objects, ranging from explicit 3D models to raw point clouds. |
🔗 |
Fri 4:20 p.m. - 4:32 p.m.
|
Contributed Talk : Disentangling 3D Prototypical Networks for Few-Shot Concept Learning
(
Talk
)
SlidesLive Video » |
🔗 |
Fri 4:32 p.m. - 4:44 p.m.
|
Contributed Talk : Deep Affordance Foresight: Planning for What Can Be Done Next
(
Talk
)
SlidesLive Video » |
🔗 |
Fri 4:44 p.m. - 4:56 p.m.
|
Contributed talk : Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation
(
Talk
)
SlidesLive Video » |
🔗 |
Fri 4:56 p.m. - 6:10 p.m.
|
Panel
What should be in an object representation, what should an object representation be able to do, and how do we measure and compare them? Join us for a panel discussion with Wilka Carvalho, Judy Fan, Tejas Kulkarni, and Chris Xie, moderated by Rachit Dubey. |
· Wilka Carvalho · Judith Fan · Tejas Kulkarni · Christopher Xie 🔗 |
Fri 6:10 p.m. - 6:15 p.m.
|
Concluding Remarks
|
🔗 |
Fri 6:15 p.m. - 7:15 p.m.
|
Poster Session B in GatherTown ( Poster Session ) link » | 🔗 |
Author Information
William Agnew (University of Washington)
Rim Assouel (umontreal.ca)
Michael Chang (University of California, Berkeley)
Ph.D. student at Berkeley AI Research, U.C. Berkeley B.S. in Computer Science from MIT Former research intern under Juergen Schmidhuber, Istituto Dalle Molle di Studi sull'Intelligenza Artificiale (IDSIA) Former undergraduate researcher under Joshua Tenenbaum and Antonio Torralba, MIT
Antonia Creswell (Deep Mind)
Antonia Creswell is a Senior Research Scientist at DeepMind in the Cognition team. Her work focuses on the learning and integration of object representations in dynamic models. She completed her PhD on representation learning at Imperial College London in the department of Bioengineering.
Eliza Kosoy (UC Berkeley)
Working with Alison Gopnik (and a few others at MIT and Berkeley) on the intersections of ai and child development
Aravind Rajeswaran (University of Washington)
Sjoerd van Steenkiste (Dalle Molle Institute for Artificial Intelligence Research (IDSIA))
More from the Same Authors
-
2021 : CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery »
Misha Laskin · Hao Liu · Xue Bin Peng · Denis Yarats · Aravind Rajeswaran · Pieter Abbeel -
2021 : Exploring through Random Curiosity with General Value Functions »
Aditya Ramesh · Louis Kirsch · Sjoerd van Steenkiste · Jürgen Schmidhuber -
2021 : Unsupervised Learning of Temporal Abstractions using Slot-based Transformers »
Anand Gopalakrishnan · Kazuki Irie · Jürgen Schmidhuber · Sjoerd van Steenkiste -
2021 : Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL »
Catherine Cang · Aravind Rajeswaran · Pieter Abbeel · Misha Laskin -
2021 : Rebuilding Trust: Queer in AI Approach to Artificial Intelligence Risk Management »
William Agnew · Arjun Subramonian · Juan Pajaro Velasquez · Ashwin S -
2022 : Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement »
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang -
2022 : Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement »
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang -
2022 : Solving Math Word Problems with Process-based and Outcome-based Feedback »
Jonathan Uesato · Nate Kushman · Ramana Kumar · H. Francis Song · Noah Siegel · Lisa Wang · Antonia Creswell · Geoffrey Irving · Irina Higgins -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 : Real World Offline Reinforcement Learning with Realistic Data Source »
Gaoyue Zhou · Liyiming Ke · Siddhartha Srinivasa · Abhinav Gupta · Aravind Rajeswaran · Vikash Kumar -
2022 : Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement »
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang -
2022 : Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement »
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang -
2022 : Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement »
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang -
2022 : Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement »
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang -
2022 : Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement »
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang -
2022 : Train Offline, Test Online: A Real Robot Learning Benchmark »
Gaoyue Zhou · Victoria Dean · Mohan Kumar Srirama · Aravind Rajeswaran · Jyothish Pari · Kyle Hatch · Aryan Jain · Tianhe Yu · Pieter Abbeel · Lerrel Pinto · Chelsea Finn · Abhinav Gupta -
2022 : Real World Offline Reinforcement Learning with Realistic Data Source »
Gaoyue Zhou · Liyiming Ke · Siddhartha Srinivasa · Abhinav Gupta · Aravind Rajeswaran · Vikash Kumar -
2023 Poster: Im-Promptu: In-Context Composition from Image Prompts »
Bhishma Dedhia · Michael Chang · Jake Snell · Tom Griffiths · Niraj Jha -
2022 : Solving Math Word Problems with Process-based and Outcome-based Feedback »
Jonathan Uesato · Nate Kushman · Ramana Kumar · H. Francis Song · Noah Siegel · Lisa Wang · Antonia Creswell · Geoffrey Irving · Irina Higgins -
2022 : Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement »
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang -
2022 Poster: Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation »
Michael Chang · Tom Griffiths · Sergey Levine -
2021 Poster: Visual Adversarial Imitation Learning using Variational Models »
Rafael Rafailov · Tianhe Yu · Aravind Rajeswaran · Chelsea Finn -
2021 Poster: COMBO: Conservative Offline Model-Based Policy Optimization »
Tianhe Yu · Aviral Kumar · Rafael Rafailov · Aravind Rajeswaran · Sergey Levine · Chelsea Finn -
2021 Poster: Decision Transformer: Reinforcement Learning via Sequence Modeling »
Lili Chen · Kevin Lu · Aravind Rajeswaran · Kimin Lee · Aditya Grover · Misha Laskin · Pieter Abbeel · Aravind Srinivas · Igor Mordatch -
2021 Affinity Workshop: Queer in AI Workshop 2 »
Claas Voelcker · Arjun Subramonian · Vishakha Agrawal · Luca Soldaini · Pan Xu · Pranav A · William Agnew · Juan Pajaro Velasquez · Yanan Long · Sharvani Jha · Ashwin S · Mary Anne Smart · Patrick Feeney · Ruchira Ray -
2021 Poster: Reinforcement Learning with Latent Flow »
Wenling Shang · Xiaofei Wang · Aravind Srinivas · Aravind Rajeswaran · Yang Gao · Pieter Abbeel · Misha Laskin -
2021 Poster: Unsupervised Object-Based Transition Models For 3D Partially Observable Environments »
Antonia Creswell · Rishabh Kabra · Chris Burgess · Murray Shanahan -
2021 Poster: SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition »
Rishabh Kabra · Daniel Zoran · Goker Erdogan · Loic Matthey · Antonia Creswell · Matt Botvinick · Alexander Lerchner · Chris Burgess -
2021 Affinity Workshop: Queer in AI Workshop 1 »
Claas Voelcker · Arjun Subramonian · Vishakha Agrawal · Luca Soldaini · Pan Xu · Pranav A · William Agnew · Juan Pajaro Velasquez · Yanan Long · Sharvani Jha · Ashwin S · Mary Anne Smart · Patrick Feeney · Ruchira Ray -
2021 : Balancing Structure In NeuroSymbolic Methods »
Antonia Creswell -
2021 Tutorial: Pay Attention to What You Need: Do Structural Priors Still Matter in the Age of Billion Parameter Models? »
Irina Higgins · Antonia Creswell · Sébastien Racanière -
2020 : Introduction »
William Agnew -
2020 Workshop: Resistance AI Workshop »
Suzanne Kite · Mattie Tesfaldet · J Khadijah Abdurahman · William Agnew · Elliot Creager · Agata Foryciarz · Raphael Gontijo Lopes · Pratyusha Kalluri · Marie-Therese Png · Manuel Sabin · Maria Skoularidou · Ramon Vilarino · Rose Wang · Sayash Kapoor · Micah Carroll -
2020 : Laura Montoya Q&A »
Laura Montoya · William Agnew -
2019 : Poster Session »
Ethan Harris · Tom White · Oh Hyeon Choung · Takashi Shinozaki · Dipan Pal · Katherine L. Hermann · Judy Borowski · Camilo Fosco · Chaz Firestone · Vijay Veerabadran · Benjamin Lahner · Chaitanya Ryali · Fenil Doshi · Pulkit Singh · Sharon Zhou · Michel Besserve · Michael Chang · Anelise Newman · Mahesan Niranjan · Jonathon Hare · Daniela Mihai · Marios Savvides · Simon Kornblith · Christina M Funke · Aude Oliva · Virginia de Sa · Dmitry Krotov · Colin Conwell · George Alvarez · Alex Kolchinski · Shengjia Zhao · Mitchell Gordon · Michael Bernstein · Stefano Ermon · Arash Mehrjou · Bernhard Schölkopf · John Co-Reyes · Michael Janner · Jiajun Wu · Josh Tenenbaum · Sergey Levine · Yalda Mohsenzadeh · Zhenglong Zhou -
2019 : Spotlights 1 »
Michael Chang · Jan Chorowski · Matthew Dirks -
2019 Poster: Are Disentangled Representations Helpful for Abstract Visual Reasoning? »
Sjoerd van Steenkiste · Francesco Locatello · Jürgen Schmidhuber · Olivier Bachem -
2019 Poster: MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies »
Xue Bin Peng · Michael Chang · Grace Zhang · Pieter Abbeel · Sergey Levine -
2018 : Poster Session 1 + Coffee »
Tom Van de Wiele · Rui Zhao · J. Fernando Hernandez-Garcia · Fabio Pardo · Xian Yeow Lee · Xiaolin Andy Li · Marcin Andrychowicz · Jie Tang · Suraj Nair · Juhyeon Lee · Cédric Colas · S. M. Ali Eslami · Yen-Chen Wu · Stephen McAleer · Ryan Julian · Yang Xue · Matthia Sabatelli · Pranav Shyam · Alexandros Kalousis · Giovanni Montana · Emanuele Pesce · Felix Leibfried · Zhanpeng He · Chunxiao Liu · Yanjun Li · Yoshihide Sawada · Alexander Pashevich · Tejas Kulkarni · Keiran Paster · Luca Rigazio · Quan Vuong · Hyunggon Park · Minhae Kwon · Rivindu Weerasekera · Shamane Siriwardhanaa · Rui Wang · Ozsel Kilinc · Keith Ross · Yizhou Wang · Simon Schmitt · Thomas Anthony · Evan Cater · Forest Agostinelli · Tegg Sung · Shirou Maruyama · Alexander Shmakov · Devin Schwab · Mohammad Firouzi · Glen Berseth · Denis Osipychev · Jesse Farebrother · Jianlan Luo · William Agnew · Peter Vrancx · Jonathan Heek · Catalin Ionescu · Haiyan Yin · Megumi Miyashita · Nathan Jay · Noga H. Rotman · Sam Leroux · Shaileshh Bojja Venkatakrishnan · Henri Schmidt · Jack Terwilliger · Ishan Durugkar · Jonathan Sauder · David Kas · Arash Tavakoli · Alain-Sam Cohen · Philip Bontrager · Adam Lerer · Thomas Paine · Ahmed Khalifa · Ruben Rodriguez · Avi Singh · Yiming Zhang -
2018 : Lunch & Posters »
Haytham Fayek · German Parisi · Brian Xu · Pramod Kaushik Mudrakarta · Sophie Cerf · Sarah Wassermann · Davit Soselia · Rahaf Aljundi · Mohamed Elhoseiny · Frantzeska Lavda · Kevin J Liang · Arslan Chaudhry · Sanmit Narvekar · Vincenzo Lomonaco · Wesley Chung · Michael Chang · Ying Zhao · Zsolt Kira · Pouya Bashivan · Banafsheh Rafiee · Oleksiy Ostapenko · Andrew Jones · Christos Kaplanis · Sinan Kalkan · Dan Teng · Xu He · Vincent Liu · Somjit Nath · Sungsoo Ahn · Ting Chen · Shenyang Huang · Yash Chandak · Nathan Sprague · Martin Schrimpf · Tony Kendall · Jonathan Richard Schwarz · Michael Li · Yunshu Du · Yen-Chang Hsu · Samira Abnar · Bo Wang -
2017 : Relational neural expectation maximization »
Sjoerd van Steenkiste -
2017 Poster: Towards Generalization and Simplicity in Continuous Control »
Aravind Rajeswaran · Kendall Lowrey · Emanuel Todorov · Sham Kakade -
2017 Poster: Neural Expectation Maximization »
Klaus Greff · Sjoerd van Steenkiste · Jürgen Schmidhuber