BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies

We ask whether AI agents can reach the conclusions of a published biomedical study from its data alone. Existing evaluations do not consistently separate analysis from prior knowledge or retrieval of the published answer. We introduce BioStudyBench, a benchmark of 26 long-horizon analysis tasks drawn from studies first published between July and September 2026, after the knowledge cutoffs of the models we evaluate, semi-automatically filtered down from 404,019 PubMed records. In each task, the agent receives a neutral research question but no data files, so it must find and download the relevant public data, search the literature through tools that return only records dated before its cutoff, and report findings through data analysis. To separate analysis from prior knowledge, we run every task both with and without access to the data. Across eight models, access to the data and tools raises the pass rate by 47 percentage points on average over prior knowledge alone, showing that models cannot complete the benchmark from what they already know. Open-weight models across sizes trail closed-weight models, with the best open-weight model passing 81.3% of tasks against 97.2% for the best closed-weight model.