Training Language Models to Negotiate in the Game of Diplomacy

Despite much progress in training AI systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. I will describe how we adapted language models to negotiate with people, reaching human-level performance in Diplomacy. A typical game involves generating hundreds of messages, which must be grounded in the game state, dialogue history, and the agent’s intended actions - all in a domain far from the pre-training data. The core of our approach is a method for linking language models to a symbolic planning module. Across 40 games of an anonymous online Diplomacy league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.