Timezone: »
Poster
Continual Learning with Evolving Class Ontologies
Zhiqiu Lin · Deepak Pathak · Yu-Xiong Wang · Deva Ramanan · Shu Kong
Lifelong learners must recognize concept vocabularies that evolve over time. A common yet underexplored scenario is learning with class labels that continually refine/expand old classes. For example, humans learn to recognize ${\tt dog}$ before dog breeds. In practical settings, dataset ${\it versioning}$ often introduces refinement to ontologies, such as autonomous vehicle benchmarks that refine a previous ${\tt vehicle}$ class into ${\tt school-bus}$ as autonomous operations expand to new cities. This paper formalizes a protocol for studying the problem of ${\it Learning with Evolving Class Ontology}$ (LECO). LECO requires learning classifiers in distinct time periods (TPs); each TP introduces a new ontology of "fine" labels that refines old ontologies of "coarse" labels (e.g., dog breeds that refine the previous ${\tt dog}$). LECO explores such questions as whether to annotate new data or relabel the old, how to exploit coarse labels, and whether to finetune the previous TP's model or train from scratch. To answer these questions, we leverage insights from related problems such as class-incremental learning. We validate them under the LECO protocol through the lens of image classification (on CIFAR and iNaturalist) and semantic segmentation (on Mapillary). Extensive experiments lead to some surprising conclusions; while the current status quo in the field is to relabel existing datasets with new class ontologies (such as COCO-to-LVIS or Mapillary1.2-to-2.0), LECO demonstrates that a far better strategy is to annotate ${\it new}$ data with the new ontology. However, this produces an aggregate dataset with inconsistent old-vs-new labels, complicating learning. To address this challenge, we adopt methods from semi-supervised and partial-label learning. We demonstrate that such strategies can surprisingly be made near-optimal, in the sense of approaching an "oracle" that learns on the aggregate dataset exhaustively labeled with the newest ontology.
Author Information
Zhiqiu Lin (Carnegie Mellon University)
Deepak Pathak (Carnegie Mellon University)
Yu-Xiong Wang (School of Computer Science, Carnegie Mellon University)
Deva Ramanan (Carnegie Mellon University)
Shu Kong (Texas A&M University)
More from the Same Authors
-
2021 Spotlight: ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction »
Gengshan Yang · Deqing Sun · Varun Jampani · Daniel Vlasic · Forrester Cole · Ce Liu · Deva Ramanan -
2021 : Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting »
Benjamin Wilson · William Qi · Tanmay Agarwal · John Lambert · Jagjeet Singh · Siddhesh Khandelwal · Bowen Pan · Ratnesh Kumar · Andrew Hartnett · Jhony Kaesemodel Pontes · Deva Ramanan · Peter Carr · James Hays -
2021 : RB2: Robotic Manipulation Benchmarking with a Twist »
Sudeep Dasari · Jianren Wang · Joyce Hong · Shikhar Bahl · Yixin Lin · Austin Wang · Abitha Thankaraj · Karanbir Chahal · Berk Calli · Saurabh Gupta · David Held · Lerrel Pinto · Deepak Pathak · Vikash Kumar · Abhinav Gupta -
2021 : The CLEAR Benchmark: Continual LEArning on Real-World Imagery »
Zhiqiu Lin · Jia Shi · Deepak Pathak · Deva Ramanan -
2021 : Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives »
Murtaza Dalal · Deepak Pathak · Russ Salakhutdinov -
2022 : Test-time adaptation with slot-centric models »
Mihir Prabhudesai · Sujoy Paul · Sjoerd van Steenkiste · Mehdi S. M. Sajjadi · Anirudh Goyal · Deepak Pathak · Katerina Fragkiadaki · Gaurav Aggarwal · Thomas Kipf -
2022 : Test-time adaptation with slot-centric models »
Mihir Prabhudesai · Sujoy Paul · Sjoerd van Steenkiste · Mehdi S. M. Sajjadi · Anirudh Goyal · Deepak Pathak · Katerina Fragkiadaki · Gaurav Aggarwal · Thomas Kipf -
2022 : The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalence »
Brando Miranda · Patrick Yu · Yu-Xiong Wang · Sanmi Koyejo -
2023 Poster: HASSOD: Hierarchical Adaptive Self-Supervised Object Detection »
Shengcao Cao · Dhiraj Joshi · Liangyan Gui · Yu-Xiong Wang -
2023 Poster: Generating and Distilling Discrete Adversarial Examples from Large-Scale Models »
Andy Zhou · Jindong Wang · Yu-Xiong Wang · Haohan Wang -
2023 Poster: Test time Adaptation with Diffusion Models »
Mihir Prabhudesai · Tsung-Wei Ke · Alex Li · Deepak Pathak · Katerina Fragkiadaki -
2023 Poster: PyNeRF: Pyramidal Neural Radiance Fields »
Haithem Turki · Michael Zollhöfer · Christian Richardt · Deva Ramanan -
2023 Poster: ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields »
Jiahua Dong · Yu-Xiong Wang -
2023 Poster: A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories »
Kai Yan · Alex Schwing · Yu-Xiong Wang -
2023 Poster: A High-Resolution Dataset for Instance Detection with Multi-View Object Capture »
QIANQIAN SHEN · Yunhan Zhao · Nahyun Kwon · Jeeeun Kim · Yanan Li · Shu Kong -
2023 Poster: OV-PARTS: Towards Open-Vocabulary Part Segmentation »
Meng Wei · Xiaoyu Yue · Wenwei Zhang · Shu Kong · Xihui Liu · Jiangmiao Pang -
2023 Poster: YouTubePD: A Multimodal Benchmark for Parkinson’s Disease Analysis »
Andy Zhou · Samuel Li · Pranav Sriram · Xiang Li · Jiahua Dong · Ansh Sharma · Yuanyi Zhong · Shirui Luo · Volodymyr Kindratenko · George Heintz · Christopher Zallek · Yu-Xiong Wang -
2022 Poster: CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations »
Kai Yan · Alex Schwing · Yu-Xiong Wang -
2022 Poster: Learning to Discover and Detect Objects »
Vladimir Fomenko · Ismail Elezi · Deva Ramanan · Laura Leal-Taixé · Aljosa Osep -
2021 Oral: Interesting Object, Curious Agent: Learning Task-Agnostic Exploration »
Simone Parisi · Victoria Dean · Deepak Pathak · Abhinav Gupta -
2021 Poster: Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives »
Murtaza Dalal · Deepak Pathak · Russ Salakhutdinov -
2021 Poster: Discovering and Achieving Goals via World Models »
Russell Mendonca · Oleh Rybkin · Kostas Daniilidis · Danijar Hafner · Deepak Pathak -
2021 Poster: Functional Regularization for Reinforcement Learning via Learned Fourier Features »
Alex Li · Deepak Pathak -
2021 Poster: ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction »
Gengshan Yang · Deqing Sun · Varun Jampani · Daniel Vlasic · Forrester Cole · Ce Liu · Deva Ramanan -
2021 Poster: Interesting Object, Curious Agent: Learning Task-Agnostic Exploration »
Simone Parisi · Victoria Dean · Deepak Pathak · Abhinav Gupta -
2021 Poster: NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild »
Jason Zhang · Gengshan Yang · Shubham Tulsiani · Deva Ramanan -
2020 Poster: Neural Dynamic Policies for End-to-End Sensorimotor Learning »
Shikhar Bahl · Mustafa Mukadam · Abhinav Gupta · Deepak Pathak -
2020 Spotlight: Neural Dynamic Policies for End-to-End Sensorimotor Learning »
Shikhar Bahl · Mustafa Mukadam · Abhinav Gupta · Deepak Pathak -
2020 Session: Orals & Spotlights Track 14: Reinforcement Learning »
Deepak Pathak · Martha White -
2020 Poster: Sparse Graphical Memory for Robust Planning »
Scott Emmons · Ajay Jain · Misha Laskin · Thanard Kurutach · Pieter Abbeel · Deepak Pathak -
2019 Poster: Volumetric Correspondence Networks for Optical Flow »
Gengshan Yang · Deva Ramanan -
2017 Poster: Learning to Model the Tail »
Yu-Xiong Wang · Deva Ramanan · Martial Hebert -
2017 Poster: Attentional Pooling for Action Recognition »
Rohit Girdhar · Deva Ramanan