Inferring cognitive strategies from groups of animals in natural environments

Understanding multi-agent behavior in natural environments requires world models that jointly capture agent interactions, environmental structure, and the cognitive strategies that shape collective dynamics. Existing approaches typically focus on isolated components — such as tracking, simulation, or inference — without integration into a unified pipeline for reasoning about relationships between behavior, environment, and cognition. Here, we introduce new behavioral data from groups of birds foraging in outdoor 3D environments, as well as an open-source framework for interpreting this data. The framework combines detection and tracking of multiple animals, semantic 3D environment reconstruction, multi-agent simulation, and graph neural network–based inference. By bringing together behavior, context, and predictive modeling, this paper lays the groundwork for investigating latent cognitive strategies across species and environments, and identifies key challenges to interpreting models of these systems.