Conclusion Fidelity of AI Agents in Synthetic Omics Environments

Synthetic data is proposed as a basis for discovery in biomedicine, and AI agents are beginning to carry out the analysis. Synthetic data is accepted on the strength of fidelity metrics, which measure its similarity to real data. However, similarity does not establish that an agent analysing synthetic data reaches the conclusions it would reach on the real data. We build synthetic molecular breast cancer cohorts into which we introduce flaws of known location and strength, ask agents built on three open-weight models to analyse each cohort without a fixed plan, and check every claim they make against the real cohort. Moderately imperfect generators yield claims that almost all hold on the real cohort but miss real findings. A weakened true relationship disappears from the agents' conclusions rather than being reversed, while a fabricated signal is reported as a marker in most runs once it approaches the strength of genuine markers. When a fabrication contradicts textbook biology, the agent notices the conflict but rarely rejects it. Standard fidelity metrics register neither flaw. Re-evaluating each claim on an ensemble of generators separates claims that hold on real data from artefacts, unless all generators share the flaw.