Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status. Leaderboards typically publish the model and the score, so capability and leakage remain observationally equivalent. Existing taxonomies classify contamination for automated detection; they do not answer the question a reporter faces at publication: given the mitigations already applied, which validity threats remain open? We introduce a taxonomy organized by the mitigation each type defeats — direct, derivative, temporal, distributional, and acquired — spanning training-time and evaluation-time leakage. Holding out a private test set closes the first of these alone. The fifth is acquired during the evaluation itself; because it is a property of one run, it must be recorded with the reported score rather than with the benchmark release. We operationalize the taxonomy as a four-field disclosure protocol in which "unknown" is a valid entry, released under CC BY 4.0 with a JSON Schema, a validator, and worked examples. Two coders external to the design team applied a pre-registered instrument to 41 documents. Per-variable linear-weighted 𝜅 runs from 0.00 to 0.35 (median 0.21) over 29 main-pass documents against a single-coder test-retest ceiling of 0.84, collapsing under the class skew the registration anticipated; pooling raises 𝜅 to 0.46 through chance correction rather than better agreement, so we report per variable. Of the five variables where no single code dominates, three clear the prevalence-robust threshold registered in advance; the two that fall below it are strata reporting and the acquired type introduced here. Disagreement concentrates on when a variable applies rather than on what a document states. The audit also confirms the registered prediction that elicitation and regeneration each stay under 25% in every stratum: elicitation budgets are reported in 13% of documents, and no document addresses all five types. The contribution is the taxonomy, the score-side artifact that follows from it, and a pre-registered measurement of both instrument reliability and current disclosure.