Repeated-run reliability of biomedical agents: measurement pitfalls and selective deferral

Biomedical agents often answer differently when rerun on the same task. We evaluated four biomedical-agent configurations (Biomni, GenoMAS, AutoBA, Open-Rosalind) on their respective benchmarks (13–118 tasks, 4 runs each), distinguishing pre-registered, pre-specified and exploratory results. The tested configurations differ primarily in how they fail: AutoBA answers 46% of runs but is right on 88% of those; GenoMAS answers 93% and is right on 40%. Under grader-aligned, answer-only equivalence with failed runs never agreeing, agreement among Biomni’s runs predicts correctness (run-level AUROC 0.84) but weakly within a task (0.65 [0.51, 0.79]), mostly reflecting task difficulty. Failure handling reversed AutoBA’s conclusion, and answer-key granularity changed GenoMAS’s AUROC from uninformative by construction to 0.734 (post hoc key); we give a reporting checklist. In an exploratory archive replay, two-run agreement cut Biomni’s error from 31% to 13.3% at 41.5% coverage, for 2.0× the agent tokens. Pre-registered tests on 2,616 fresh runs (100 previously unused tasks) confirmed within-task discrimination (0.705) and that agreement with one retry cut error by 15.7 points, from 30.4% (single run) to 14.7%, at 56.7% coverage. Deferral did not clearly help GenoMAS, and none of three correction approaches robustly outperformed plurality voting. A pre-registered checker, an off-the-shelf LLM reading a finished Biomni run’s log up to its answer, separated correct from incorrect runs within tasks (AUROC 0.76; 0.74 on fresh runs); at approximately matched coverage it was non-inferior to two-run agreement, using half the agent tokens plus a checker pass; transfer to another model was inconclusive. These findings support selective deferral for cutting accepted-answer errors in the tested configurations, though measurement matters and generalization is limited.