Seven Instrument Defects Found in a Synthetic Benchmark for Data-Analysis Agents, and the Wrong Answers That Remain

This case report concerns a synthetic benchmark for language-model agents that run statistical analyses. Every task outside one trap-free family has a correct value and a trap, the number that one named analytical mistake produces. Language models working through the agent tool Claude Code built the benchmark and drafted this paper. They found seven defects in its questions, harness and scorer, as this paper divides them, each by reading, or trying to read, what the evaluated models produced rather than from an aggregate score; reading missed one in the first run that showed it. The defects do not account for every wrong answer: in the last of five versions the four models are still not correct in 23 to 98 of 120 rollouts (attempts) each; 22 of the 23 of Qwen3-8B are T1 answers scored on rows held out from the fit, which the question may allow. We withdraw an earlier manuscript's claim that taking the trap is a property of the model: its trap events all lie in a family whose question was then defective, and over two runs 8 of 30 rollouts of Qwen3-8B on that family are trap events against 1 of 30 for Qwen2.5-7B-Instruct (p = 0.026, chosen after seeing the counts; Holm-adjusted 0.23 or 0.26, not below 0.05); in the last run only Phi-3-mini takes that trap, and whether trap-taking depends on the model is open. On this case alone, we recommend reading a trap-free family first, pairing each zero count with a positive control and storing a transcript with each score. The first author has since read every transcript, reviewed the pipeline code and spot-checked the score files and counts; the code, data and transcripts are not released.