Timezone: »

 
Poster
Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs
Jinguo Zhu · Xizhou Zhu · Wenhai Wang · Xiaohua Wang · Hongsheng Li · Xiaogang Wang · Jifeng Dai

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #632
in Poster Session 4 »

To build an artificial neural network like the biological intelligence system, recent works have unified numerous tasks into a generalist model, which can process various tasks with shared parameters and do not have any task-specific modules. While generalist models achieve promising results on various benchmarks, they have performance degradation on some tasks compared with task-specialized models. In this work, we find that interference among different tasks and modalities is the main factor to this phenomenon. To mitigate such interference, we introduce the Conditional Mixture-of-Experts (Conditional MoEs) to generalist models. Routing strategies under different levels of conditions are proposed to take both the training/inference cost and generalization ability into account. By incorporating the proposed Conditional MoEs, the recently proposed generalist model Uni-Perceiver can effectively mitigate the interference across tasks and modalities, and achieves state-of-the-art results on a series of downstream tasks via prompt tuning on 1% of downstream data. Moreover, the introduction of Conditional MoEs still holds the generalization ability of generalist models to conduct zero-shot inference on new tasks, e.g., videotext retrieval and video caption. Code and pre-trained generalist models are publicly released at https://github.com/fundamentalvision/Uni-Perceiver.

Keywords: Task Interference · mixture of experts · Generalist Models
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Jinguo Zhu (Xi'an Jiaotong University)
Xizhou Zhu (Shanghai AI Laboratory)
Wenhai Wang (Nanjing University)
Xiaohua Wang (Xi'an Jiaotong University)
Hongsheng Li (The Chinese University of Hong Kong)
Xiaogang Wang (The Chinese University of Hong Kong)
Jifeng Dai (Tsinghua University)

More from the Same Authors