Controlling World Model Generation for Imagined Curriculum Learning

Recent work in Unsupervised Environment Design (UED) has examined automatically generating training curricula for policies trained using deep reinforcement learning, with the aim of improving generalisation, and zero-shot capabilities. However, training curriculum-generating adversaries still poses a challenging problem. Existing work has examined online environment generation, with the aim of reducing the difficulty of the adversary credit-assignment problem, but this relies on complex, handcrafted environment simulators that allow for dynamic environment generation. In this work, we instead introduce a world model architecture that allows for an adversarial teacher to control world generation online, through Dynamically-controlled Model Generation (DyMGen), as a student explores the environment. This allows for these complex, handcrafted simulators to be replaced with learned models. In this work, we show that agents trained in these adversary-controlled world models significantly outperform curriculum-free baselines, whilst also approaching the performance of adversarially-trained agents with full access to the true environment.