Timezone: »

 
Poster
Weak-shot Semantic Segmentation via Dual Similarity Transfer
Junjie Chen · Li Niu · Siyuan Zhou · Jianlou Si · Chen Qian · Liqing Zhang

@

Semantic segmentation is a practical and active task, but severely suffers from the expensive cost of pixel-level labels when extending to more classes in wider applications. To this end, we focus on the problem named weak-shot semantic segmentation, where the novel classes are learnt from cheaper image-level labels with the support of base classes having off-the-shelf pixel-level labels. To tackle this problem, we propose a dual similarity transfer framework, which is built upon MaskFormer to disentangle the semantic segmentation task into single-label classification and binary segmentation for each proposal. Specifically, the binary segmentation sub-task allows proposal-pixel similarity transfer from base classes to novel classes, which enables the mask learning of novel classes. We also learn pixel-pixel similarity from base classes and distill such class-agnostic semantic similarity to the semantic masks of novel classes, which regularizes the segmentation model with pixel-level semantic relationship across images. In addition, we propose a complementary loss to facilitate the learning of novel classes. Comprehensive experiments on the challenging COCO-Stuff-10K and ADE20K datasets demonstrate the effectiveness of our method.

[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Junjie Chen (Shanghai Jiao Tong University)
Li Niu (Shanghai Jiao Tong University)
Siyuan Zhou (Shanghai Jiao Tong University)
Jianlou Si
Chen Qian (SenseTime)
Liqing Zhang (Shanghai Jiao Tong University)

More from the Same Authors