Executable Is Not Faithful: Molecular Identity and Quantity Binding in Scientific Tool Calls
Abstract
A scientifically faithful tool call must preserve molecular identity, data provenance, arithmetic direction and units. We test these contracts with 128 FreeSolv molecules, paired canonical and atom-permuted SMILES, and two frozen language models. Each model selects among four molecular records for an experimental lookup or a calculated-minus-experimental contrast; a deterministic tool performs the exact database arithmetic and conversion. Of 1,024 schema-constrained generations, 1,024 are executable and 670 are executable but contract-wrong. Equivalent SMILES rendering changes wrong-call rates by Qwen3: +42.19, +34.38 percentage points; SmolLM2: +3.91, +0.78 percentage points for the two tasks, respectively; 2 of four multiplicity-adjusted intervals exclude zero. SmolLM2's contrast task is already near-complete failure at baseline. Literal SMILES validation falsely rejects 93 faithful calls, while molecular-formula validation admits 15 incorrect calls. Graph-and-quantity validation blocks the observed errors through abstention, without repairing them. A no-LLM graph resolver solves the structured binding problem directly. The contribution is an auditable scientific-interface benchmark and a separation of execution from request fidelity, not a new chemistry model or discovery result.