Timezone: »
We tackle the problem of novel class discovery and localization (NCDL). In this setting, we assume a source dataset with supervision for only some object classes. Instances of other classes need to be discovered, classified, and localized automatically based on visual similarity without any human supervision. To tackle NCDL, we propose a two-stage object detection network Region-based NCDL (RNCDL) that uses a region proposal network to localize regions of interest (RoIs). We then train our network to learn to classify each RoI, either as one of the known classes, seen in the source dataset, or one of the novel classes, with a long-tail distribution constraint on the class assignments, reflecting the natural frequency of classes in the real world. By training our detection network with this objective in an end-to-end manner, it learns to classify all region proposals for a large variety of classes, including those not part of the labeled object class vocabulary. Our experiments conducted using COCO and LVIS datasets reveal that our method is significantly more effective than multi-stage pipelines that rely on traditional clustering algorithms. Furthermore, we demonstrate the generality of our approach by applying our method to a large-scale Visual Genome dataset, where our network successfully learns to detect various semantic classes without direct supervision.
Author Information
Vladimir Fomenko (Technical University of Munich)

I currently conduct applied research and machine learning engineering for large language models as an applied scientist in Azure AI. Prior to this, I completed my master's degree in Artificial Intelligence from Technical University of Munich. My research interests are in natural language processing, computer vision, and representation learning, using advanced deep learning methods.
Ismail Elezi (Technical University of Munich)
Deva Ramanan (Carnegie Mellon University)
Laura Leal-Taixé (TUM)
Aljosa Osep (TU Munich)
More from the Same Authors
-
2021 Spotlight: ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction »
Gengshan Yang · Deqing Sun · Varun Jampani · Daniel Vlasic · Forrester Cole · Ce Liu · Deva Ramanan -
2021 : Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting »
Benjamin Wilson · William Qi · Tanmay Agarwal · John Lambert · Jagjeet Singh · Siddhesh Khandelwal · Bowen Pan · Ratnesh Kumar · Andrew Hartnett · Jhony Kaesemodel Pontes · Deva Ramanan · Peter Carr · James Hays -
2021 : The CLEAR Benchmark: Continual LEArning on Real-World Imagery »
Zhiqiu Lin · Jia Shi · Deepak Pathak · Deva Ramanan -
2021 : STEP: Segmenting and Tracking Every Pixel »
Mark Weber · Jun Xie · Maxwell Collins · Yukun Zhu · Paul Voigtlaender · Hartwig Adam · Bradley Green · Andreas Geiger · Bastian Leibe · Daniel Cremers · Aljosa Osep · Laura Leal-Taixé · Liang-Chieh Chen -
2021 : DENETHOR: The DynamicEarthNET dataset for Harmonized, inter-Operable, analysis-Ready, daily crop monitoring from space »
Lukas Kondmann · Aysim Toker · Marc Rußwurm · Andrés Camero · Devis Peressuti · Grega Milcinski · Pierre-Philippe Mathieu · Nicolas Longepe · Timothy Davis · Giovanni Marchisio · Laura Leal-Taixé · Xiaoxiang Zhu -
2022 : PolarMOT: How Far Can Geometric Relations Take Us in 3D Multi-Object Tracking? »
Aleksandr Kim · Guillem Braso · Aljosa Osep · Laura Leal-Taixé -
2022 : PolarMOT: How far can geometric relations take us in 3D multi-object tracking? »
Aleksandr Kim · Guillem Braso · Aljosa Osep · Laura Leal-Taixé -
2023 Poster: PyNeRF: Pyramidal Neural Radiance Fields »
Haithem Turki · Michael Zollhöfer · Christian Richardt · Deva Ramanan -
2022 Poster: Continual Learning with Evolving Class Ontologies »
Zhiqiu Lin · Deepak Pathak · Yu-Xiong Wang · Deva Ramanan · Shu Kong -
2022 Poster: Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking? »
Patrick Dendorfer · Vladimir Yugay · Aljosa Osep · Laura Leal-Taixé -
2022 Poster: The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes »
Peter Kocsis · Peter Súkeník · Guillem Braso · Matthias Niessner · Laura Leal-Taixé · Ismail Elezi -
2021 Poster: ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction »
Gengshan Yang · Deqing Sun · Varun Jampani · Daniel Vlasic · Forrester Cole · Ce Liu · Deva Ramanan -
2021 Poster: NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild »
Jason Zhang · Gengshan Yang · Shubham Tulsiani · Deva Ramanan -
2020 Poster: Make One-Shot Video Object Segmentation Efficient Again »
Tim Meinhardt · Laura Leal-Taixé -
2020 Poster: Deep Shells: Unsupervised Shape Correspondence with Optimal Transport »
Marvin Eisenberger · Aysim Toker · Laura Leal-Taixé · Daniel Cremers -
2019 Poster: Volumetric Correspondence Networks for Optical Flow »
Gengshan Yang · Deva Ramanan -
2017 Poster: Learning to Model the Tail »
Yu-Xiong Wang · Deva Ramanan · Martial Hebert -
2017 Poster: Attentional Pooling for Action Recognition »
Rohit Girdhar · Deva Ramanan