Timezone: »

 
Poster
Multi-Agent Common Knowledge Reinforcement Learning
Christian Schroeder de Witt · Jakob Foerster · Gregory Farquhar · Philip Torr · Wendelin Boehmer · Shimon Whiteson

Wed Dec 11 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #201
in Reinforcement Learning and Planning -- Multi-Agent RL »

Cooperative multi-agent reinforcement learning often requires decentralised policies, which severely limit the agents' ability to coordinate their behaviour. In this paper, we show that common knowledge between agents allows for complex decentralised coordination. Common knowledge arises naturally in a large number of decentralised cooperative multi-agent tasks, for example, when agents can reconstruct parts of each others' observations. Since agents can independently agree on their common knowledge, they can execute complex coordinated policies that condition on this knowledge in a fully decentralised fashion. We propose multi-agent common knowledge reinforcement learning (MACKRL), a novel stochastic actor-critic algorithm that learns a hierarchical policy tree. Higher levels in the hierarchy coordinate groups of agents by conditioning on their common knowledge, or delegate to lower levels with smaller subgroups but potentially richer common knowledge. The entire policy tree can be executed in a fully decentralised fashion. As the lowest policy tree level consists of independent policies for each agent, MACKRL reduces to independently learnt decentralised policies as a special case. We demonstrate that our method can exploit common knowledge for superior performance on complex decentralised coordination tasks, including a stochastic matrix game and challenging problems in StarCraft II unit micromanagement.

Keywords: Multi-Agent RL · Hierarchical RL · Reinforcement Learning and Planning · Reinforcement Learning and Planning -> Decision and Control; Reinforcement Learning and Planning
[ Paper]  [ Poster] 

Author Information

Christian Schroeder de Witt (University of Oxford)
Jakob Foerster (Facebook AI Research)

Jakob Foerster received a CIFAR AI chair in 2019 and is starting as an Assistant Professor at the University of Toronto and the Vector Institute in the academic year 20/21. During his PhD at the University of Oxford, he helped bring deep multi-agent reinforcement learning to the forefront of AI research and interned at Google Brain, OpenAI, and DeepMind. He has since been working as a research scientist at Facebook AI Research in California, where he will continue advancing the field up to his move to Toronto. He was the lead organizer of the first Emergent Communication (EmeCom) workshop at NeurIPS in 2017, which he has helped organize ever since.

Gregory Farquhar (University of Oxford)
Philip Torr (University of Oxford)
Wendelin Boehmer (University of Oxford)
Shimon Whiteson (University of Oxford)

More from the Same Authors