Timezone: »

 
Poster
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
Muning Wen · Jakub Kuba · Runji Lin · Weinan Zhang · Ying Wen · Jun Wang · Yaodong Yang

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #520
in Poster Session 3 »

Large sequence models (SM) such as GPT series and BERT have displayed outstanding performance and generalization capabilities in natural language process, vision and recently reinforcement learning. A natural follow-up question is how to abstract multi-agent decision making also as an sequence modeling problem and benefit from the prosperous development of the SMs. In this paper, we introduce a novel architecture named Multi-Agent Transformer (MAT) that effectively casts cooperative multi-agent reinforcement learning (MARL) into SM problems wherein the objective is to map agents' observation sequences to agents' optimal action sequences. Our goal is to build the bridge between MARL and SMs so that the modeling power of modern sequence models can be unleashed for MARL. Central to our MAT is an encoder-decoder architecture which leverages the multi-agent advantage decomposition theorem to transform the joint policy search problem into a sequential decision making process; this renders only linear time complexity for multi-agent problems and, most importantly, endows MAT with monotonic performance improvement guarantee. Unlike prior arts such as Decision Transformer fit only pre-collected offline data, MAT is trained by online trial and error from the environment in an on-policy fashion. To validate MAT, we conduct extensive experiments on StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football benchmarks. Results demonstrate that MAT achieves superior performance and data efficiency compared to strong baselines including MAPPO and HAPPO. Furthermore, we demonstrate that MAT is an excellent few-short learner on unseen tasks regardless of changes in the number of agents.See our project page at https://sites.google.com/view/multi-agent-transformer.

Keywords: multi-agent reinforcement learning · transformer · sequence modeling
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Muning Wen (Shanghai Jiao Tong University)
Jakub Kuba (UC Berkeley)
Runji Lin (Institute of automation, Chinese Academy of Sciences)
Runji Lin

Runji Lin is a master's student at Institute of Automation, Chinese Academy of Sciences (CASIA). His research areas include reinforcement learning, game theory, and multi-agent system.

Weinan Zhang (Shanghai Jiao Tong University)
Ying Wen (Shanghai Jiao Tong University)
Jun Wang (UCL)
Yaodong Yang (AIG)

More from the Same Authors