`

Timezone: »

 
Poster
No-Press Diplomacy from Scratch
Anton Bakhtin · David Wu · Adam Lerer · Noam Brown

Tue Dec 07 04:30 PM -- 06:00 PM (PST) @ None #None

Prior AI successes in complex games have largely focused on settings with at most hundreds of actions at each decision point. In contrast, Diplomacy is a game with more than 10^20 possible actions per turn. Previous attempts to address games with large branching factors, such as Diplomacy, StarCraft, and Dota, used human data to bootstrap the policy or used handcrafted reward shaping. In this paper, we describe an algorithm for action exploration and equilibrium approximation in games with combinatorial action spaces. This algorithm simultaneously performs value iteration while learning a policy proposal network. A double oracle step is used to explore additional actions to add to the policy proposals. At each state, the target state value and policy for the model training are computed via an equilibrium search procedure. Using this algorithm, we train an agent, DORA, completely from scratch for a popular two-player variant of Diplomacy and show that it achieves superhuman performance. Additionally, we extend our methods to full-scale no-press Diplomacy and for the first time train an agent from scratch with no human data. We present evidence that this agent plays a strategy that is incompatible with human-data bootstrapped agents. This presents the first strong evidence of multiple equilibria in Diplomacy and suggests that self play alone may be insufficient for achieving superhuman performance in Diplomacy.

Author Information

Anton Bakhtin (Facebook AI Research)
David Wu (Facebook)
Adam Lerer (Facebook AI Research)
Noam Brown (Facebook AI Research)

More from the Same Authors

  • 2021 : A Fine-Tuning Approach to Belief State Modeling »
    Samuel Sokota · Hengyuan Hu · David Wu · Jakob Foerster · Noam Brown
  • 2021 Workshop: Cooperative AI »
    Natasha Jaques · Edward Hughes · Jakob Foerster · Noam Brown · Kalesha Bullard · Charlotte Smith
  • 2021 Poster: Scalable Online Planning via Reinforcement Learning Fine-Tuning »
    Arnaud Fickinger · Hengyuan Hu · Brandon Amos · Stuart Russell · Noam Brown
  • 2020 : Exploring generative atomic models in cryo-EM reconstruction »
    Ellen Zhong · Adam Lerer · jhdavis · Bonnie Berger
  • 2020 Poster: Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian »
    Jack Parker-Holder · Luke Metz · Cinjon Resnick · Hengyuan Hu · Adam Lerer · Alistair Letcher · Alexander Peysakhovich · Aldo Pacchiano · Jakob Foerster
  • 2020 Poster: Combining Deep Reinforcement Learning and Search for Imperfect-Information Games »
    Noam Brown · Anton Bakhtin · Adam Lerer · Qucheng Gong
  • 2019 : Contributed Talk - 3 »
    Adam Lerer
  • 2019 : Poster Session »
    Matthia Sabatelli · Adam Stooke · Amir Abdi · Paulo Rauber · Leonard Adolphs · Ian Osband · Hardik Meisheri · Karol Kurach · Johannes Ackermann · Matt Benatan · GUO ZHANG · Chen Tessler · Dinghan Shen · Mikayel Samvelyan · Riashat Islam · Murtaza Dalal · Luke Harries · Andrey Kurenkov · Konrad Żołna · Sudeep Dasari · Kristian Hartikainen · Ofir Nachum · Kimin Lee · Markus Holzleitner · Vu Nguyen · Francis Song · Christopher Grimm · Leno Silva · Yuping Luo · Yifan Wu · Alex Lee · Thomas Paine · Wei-Yang Qu · Daniel Graves · Yannis Flet-Berliac · Yunhao Tang · Suraj Nair · Matthew Hausknecht · Akhil Bagaria · Simon Schmitt · Bowen Baker · Paavo Parmas · Benjamin Eysenbach · Lisa Lee · Siyu Lin · Daniel Seita · Abhishek Gupta · Riley Simmons-Edler · Yijie Guo · Kevin Corder · Vikash Kumar · Scott Fujimoto · Adam Lerer · Ignasi Clavera Gilaberte · Nick Rhinehart · Ashvin Nair · Ge Yang · Lingxiao Wang · Sungryull Sohn · JFernando Hernandez-Garcia · Xian Yeow Lee · Rupesh Srivastava · Khimya Khetarpal · Chenjun Xiao · Luckeciano Carvalho Melo · Rishabh Agarwal · Tianhe (Kevin) Yu · Glen Berseth · Devendra Singh Chaplot · Jie Tang · Anirudh Srinivasan · Tharun Medini · Aaron Havens · Misha Laskin · Asier Mujika · Rohan Saphal · Joe Marino · Alex Ray · Joshua Achiam · Ajay Mandlekar · Zhuang Liu · Danijar Hafner · Zhiwen Tang · Ted Xiao · Michael Walton · Jeff Druce · Ferran Alet · Zhang-Wei Hong · Stephanie Chan · Anusha Nagabandi · Hao Liu · Hao Sun · Ge Liu · Dinesh Jayaraman · JD Co-Reyes · Sophia Sanborn
  • 2019 : Extended Poster Session »
    Travis LaCroix · Marie Ossenkopf · Mina Lee · Nicole Fitzgerald · Daniela Mihai · Jonathon Hare · Ali Zaidi · Alexander Cowen-Rivers · Alana Marzoev · Eugene Kharitonov · Luyao Yuan · Tomek Korbak · Paul Pu Liang · Yi Ren · Roberto Dessì · Peter Potash · Shangmin Guo · Tatsunori Hashimoto · Percy Liang · Julian Zubek · Zipeng Fu · Song-Chun Zhu · Adam Lerer
  • 2019 Poster: PyTorch: An Imperative Style, High-Performance Deep Learning Library »
    Adam Paszke · Sam Gross · Francisco Massa · Adam Lerer · James Bradbury · Gregory Chanan · Trevor Killeen · Zeming Lin · Natalia Gimelshein · Luca Antiga · Alban Desmaison · Andreas Kopf · Edward Yang · Zachary DeVito · Martin Raison · Alykhan Tejani · Sasank Chilamkurthy · Benoit Steiner · Lu Fang · Junjie Bai · Soumith Chintala
  • 2019 Poster: PHYRE: A New Benchmark for Physical Reasoning »
    Anton Bakhtin · Laurens van der Maaten · Justin Johnson · Laura Gustafson · Ross Girshick
  • 2019 Poster: Robust Multi-agent Counterfactual Prediction »
    Alexander Peysakhovich · Christian Kroer · Adam Lerer
  • 2018 : Poster Session 1 + Coffee »
    Tom Van de Wiele · Rui Zhao · JFernando Hernandez-Garcia · Fabio Pardo · Xian Yeow Lee · Xiaolin Andy Li · Marcin Andrychowicz · Jie Tang · Suraj Nair · Juhyeon Lee · Cédric Colas · Ali Eslami · Yen-Chen Wu · Stephen McAleer · Ryan Julian · Yang Xue · Matthia Sabatelli · Pranav Shyam · Alexandros Kalousis · Giovanni Montana · Emanuele Pesce · Felix Leibfried · Zhanpeng He · Chunxiao Liu · Yanjun Li · Yoshihide Sawada · Alexander Pashevich · Tejas Kulkarni · Keiran Paster · Luca Rigazio · Quan Vuong · Hyunggon Park · Minhae Kwon · Rivindu Weerasekera · Shamane Siriwardhanaa · Rui Wang · Ozsel Kilinc · Keith Ross · Yizhou Wang · Simon Schmitt · Thomas Anthony · Evan Cater · Forest Agostinelli · Tegg Sung · Shirou Maruyama · Alex Shmakov · Devin Schwab · Mohammad Firouzi · Glen Berseth · Denis Osipychev · Jesse Farebrother · Jianlan Luo · William Agnew · Peter Vrancx · Jonathan Heek · Catalin Ionescu Ionescu · Haiyan Yin · Megumi Miyashita · Nathan Jay · Noga H. Rotman · Sam Leroux · Shaileshh Bojja Venkatakrishnan · Henri Schmidt · Jack Terwilliger · Ishan Durugkar · Jonathan Sauder · David Kas · Arash Tavakoli · Alain-Sam Cohen · Philip Bontrager · Adam Lerer · Thomas Paine · Ahmed Khalifa · Rubén Rodriguez · Avi Singh · Yiming Zhang
  • 2018 Poster: Depth-Limited Solving for Imperfect-Information Games »
    Noam Brown · Tuomas Sandholm · Brandon Amos
  • 2017 Demonstration: Libratus: Beating Top Humans in No-Limit Poker »
    Noam Brown · Tuomas Sandholm
  • 2017 Poster: Safe and Nested Subgame Solving for Imperfect-Information Games »
    Noam Brown · Tuomas Sandholm
  • 2017 Oral: Safe and Nested Subgame Solving for Imperfect-Information Games »
    Noam Brown · Tuomas Sandholm
  • 2016 Workshop: Intuitive Physics »
    Adam Lerer · Jiajun Wu · Josh Tenenbaum · Emmanuel Dupoux · Rob Fergus
  • 2015 Poster: Regret-Based Pruning in Extensive-Form Games »
    Noam Brown · Tuomas Sandholm
  • 2015 Demonstration: Claudico: The World's Strongest No-Limit Texas Hold'em Poker AI »
    Noam Brown · Tuomas Sandholm