Timezone: »

 
Poster
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
Wenhai Wang · Zhe Chen · Xiaokang Chen · Jiannan Wu · Xizhou Zhu · Gang Zeng · Ping Luo · Tong Lu · Jie Zhou · Yu Qiao · Jifeng Dai

Wed Dec 13 03:00 PM -- 05:00 PM (PST) @ Great Hall & Hall B1+B2 #124
in Poster Session 4 »
Event URL: https://github.com/OpenGVLab/VisionLLM »

Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLM-based framework, our model can achieve over 60% mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The code shall be released.

Author Information

Wenhai Wang (The Chinese University of Hong Kong)
Zhe Chen (Nanjing University)
Xiaokang Chen (Peking University)
Jiannan Wu (University of Hong Kong)
Xizhou Zhu (Shanghai AI Laboratory)
Gang Zeng (Peking University)
Ping Luo (The University of Hong Kong)
Tong Lu (Nanjing University)
Jie Zhou (Tsinghua University)
Yu Qiao (Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences)
Jifeng Dai (Tsinghua University)

More from the Same Authors