CoBe: Counterfactual Benchmark for Text Editing

Counterfactual text editing is the ability to revise a given text in light of a hypothetical change while preserving facts that are not causally affected by that change. For example, when asked what would have happened if a doctor had not recommended surgery, the model output should retain the patient's medical history and modify only the consequences of the doctor's decision. This requires a causal understanding: a model must determine what events are affected by the hypothetical change and what should remain fixed. This skill is central to human reasoning, especially in safety-critical domains like healthcare, law, and business, where users may need targeted edits to existing records or documents. By contrast, the task of associational text generation generates text statistically associated with the altered condition. In the medical example, an associational response might make the patient's symptoms less severe because milder cases correlate with no surgery. Such responses produce a plausible scenario, but not the counterfactual version of the given one. Prior counterfactual-text benchmarks do not clearly separate these modes of reasoning. We introduce the Counterfactual Benchmark (CoBe), a testbed for models' zero-shot ability to counterfactually edit a given textual scenario. CoBe contains 2377 questions spanning science, human behavior, and commonsense reasoning. We test a suite of frontier and smaller models, with frontier systems achieving an average score of 53.94\%. Our findings indicate that models frequently default to associational generation rather than preserving the causal structure of the original scenario. Moreover, performance does not scale consistently with either model size or output token length, suggesting that the failure may reflect current training objectives rather than insufficient model capacity. This points to a serious reliability issue with using LLMs as Judges to evaluate tasks involving counterfactual text generation abilities.