Timezone: »
Human reasoning can be understood as a cooperation between the intuitive, associative "System-1'' and the deliberative, logical "System-2''. For existing System-1-like methods in visual activity understanding, it is crucial to integrate System-2 processing to improve explainability, generalization, and data efficiency. One possible path of activity reasoning is building a symbolic system composed of symbols and rules, where one rule connects multiple symbols, implying human knowledge and reasoning abilities.Previous methods have made progress, but are defective with limited symbols from handcraft and limited rules from visual-based annotations, failing to cover the complex patterns of activities and lacking compositional generalization. To overcome the defects, we propose a new symbolic system with two ideal important properties: broad-coverage symbols and rational rules. Collecting massive human knowledge via manual annotations is expensive to instantiate this symbolic system. Instead, we leverage the recent advancement of LLMs (Large Language Models) as an approximation of the two ideal properties, i.e., Symbols from Large Language Models (Symbol-LLM). Then, given an image, visual contents from the images are extracted andchecked as symbols and activity semantics are reasoned out based on rules via fuzzy logic calculation.Our method shows superiority in extensive activity understanding tasks. Code and data are available at https://mvig-rhos.com/symbol_llm.
Author Information
Xiaoqian Wu (Shanghai Jiao Tong University)
Yong-Lu Li (Shanghai Jiao Tong University)
Jianhua Sun (Shanghai Jiao Tong University)
Cewu Lu (Shanghai Jiao Tong University)
More from the Same Authors
-
2023 : Open X-Embodiment: Robotic Learning Datasets and RT-X Models »
Quan Vuong · Ajinkya Jain · Alex Bewley · Alexander Irpan · Alexander Khazatsky · Anant Rai · Anikait Singh · Antonin Raffin · Ayzaan Wahid · Beomjoon Kim · Bernhard Schölkopf · brian ichter · Cewu Lu · Charles Xu · Chelsea Finn · Chenfeng Xu · Cheng Chi · Chenguang Huang · Chuer Pan · Chuyuan Fu · Coline Devin · Danny Driess · Deepak Pathak · Dhruv Shah · Dieter Büchler · Dmitry Kalashnikov · Dorsa Sadigh · Edward Johns · Federico Ceola · Fei Xia · Freek Stulp · Gaoyue Zhou · Gaurav Sukhatme · Gautam Salhotra · Ge Yan · Giulio Schiavi · Hao Su · Hao-Shu Fang · Haochen Shi · Heni Ben Amor · Henrik Christensen · Hiroki Furuta · Homer Walke · Hongjie Fang · Igor Mordatch · Ilija Radosavovic · Isabel Leal · Jacky Liang · Jaehyung Kim · Jan Schneider · Jasmine Hsu · Jeannette Bohg · Jiajun Wu · Jialin Wu · Jianlan Luo · Jiayuan Gu · Jie Tan · Jitendra Malik · Jonathan Tompson · Jonathan Yang · Joseph Lim · João Silvério · Junhyek Han · Kanishka Rao · Karl Pertsch · Karol Hausman · Keegan Go · Keerthana Gopalakrishnan · Ken Goldberg · Kevin Zhang · Keyvan Majd · Krishan Rana · Krishnan Srinivasan · Lawrence Yunliang Chen · Lerrel Pinto · Liam Tan · Lionel Ott · Lisa Lee · Masayoshi TOMIZUKA · Michael Ahn · Mingyu Ding · Mohan Kumar Srirama · Mohit Sharma · Moo J Kim · Nicklas Hansen · Nicolas Heess · Nikhil Joshi · Niko Suenderhauf · Norman Di Palo · Nur Muhammad Shafiullah · Oier Mees · Oliver Kroemer · Pannag Sanketi · Paul Wohlhart · Peng Xu · Pierre Sermanet · Priya Sundaresan · Rafael Rafailov · Ran Tian · Ria Doshi · Roberto Martín-Martín · Russell Mendonca · Rutav Shah · Ryan Hoque · Ryan Julian · Samuel Bustamante · Sean Kirmani · Sergey Levine · Sherry Q Moore · Shikhar Bahl · Shivin Dass · Shuran Song · Sichun Xu · Siddhant Haldar · Simeon Adebola · Simon Guist · Soroush Nasiriany · Stefan Schaal · Stefan Welker · Stephen Tian · Sudeep Dasari · Suneel Belkhale · Takayuki Osa · Tatsuya Harada · Tatsuya Matsushima · Ted Xiao · Tianhe Yu · Tianli Ding · Todor Davchev · Tony Zhao · Trevor Darrell · Vidhi Jain · Vincent Vanhoucke · Wei Zhan · Wenxuan Zhou · Wolfram Burgard · Xi Chen · Xiaolong Wang · Xinghao Zhu · Xuanlin Li · Yao Lu · Yevgen Chebotar · Yifan Zhou · Yifeng Zhu · Yonatan Bisk · Yoonyoung Cho · Youngwoon Lee · Yuchen Cui · Yueh-Hua Wu · Yujin Tang · Yuke Zhu · Yunzhu Li · Yusuke Iwasawa · Yutaka Matsuo · Zhuo Xu · Zichen Cui · Alexander Herzog · Abhishek Padalkar · Acorn Pooley · Anthony Brohan · Ben Burgess-Limerick · Christine Chan · Jeffrey Bingham · Jihoon Oh · Kendra Byrne · Kenneth Oslund · Kento Kawaharazuka · Maximilian Du · Mingtong Zhang · Naoaki Kanazawa · Travis Armstrong · Ying Xu · Yixuan Wang · Jan Peters -
2023 : RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot »
Hao-Shu Fang · Hongjie Fang · Zhenyu Tang · Jirong Liu · Chenxi Wang · Junbo Wang · Haoyi Zhu · Cewu Lu -
2023 : RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot »
Hao-Shu Fang · Hongjie Fang · Zhenyu Tang · Jirong Liu · Chenxi Wang · Junbo Wang · Haoyi Zhu · Cewu Lu -
2023 : Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild »
Hongjie Fang · Hao-Shu Fang · Yiming Wang · Jieji Ren · Jingjing Chen · Ruo Zhang · Weiming Wang · Cewu Lu -
2022 Poster: DART: Articulated Hand Model with Diverse Accessories and Rich Textures »
Daiheng Gao · Yuliang Xiu · Kailin Li · Lixin Yang · Feng Wang · Peng Zhang · Bang Zhang · Cewu Lu · Ping Tan -
2021 Poster: Localization with Sampling-Argmax »
Jiefeng Li · Tong Chen · Ruiqi Shi · Yujing Lou · Yong-Lu Li · Cewu Lu -
2020 Poster: HOI Analysis: Integrating and Decomposing Human-Object Interaction »
Yong-Lu Li · Xinpeng Liu · Xiaoqian Wu · Yizhuo Li · Cewu Lu