Foundation models when trained at scale have shown impressive capabilities to adapt to new tasks with few examples provided in context; however, there remains a gap between the ability of these models and requirements to successfully act in embodied domains. To close this gap with reinforcement learning, our agents have to be trained at scale as well. In this talk, I will present recipes towards this end and dive into the details of how we trained AdA, utilizing a vast open ended task space, to achieve human-timescale adaptation in a 3d embodied domain. The trained agent displays on-the-fly hypothesis-driven exploration, efficient exploitation of acquired knowledge, and can successfully be prompted with first-person demonstrations.