Nothing Is Harder Than Nothing, Until You Write It Down: Absence Conventions in Stateful Tool Use
Marek Suppa ⋅ Jaroslav Kopcan
Abstract
When an agent edits persistent state through a tool, an omitted or empty field is itself an instruction, and production protocols disagree about what it instructs. JSON Merge Patch deletes a member whose value is \code{null}, and GraphQL treats the same \code{null} as a supplied value. We ask whether tool-using language models follow the update contract they are given when it contradicts the convention they have most likely learned. AbsenceBench compiles 180 semantic task families into convention-aligned and counter-conventional minimal pairs that differ in exactly one documented surface mapping, scored by an exact oracle over latent state. On 120 held-out families and five pre-registered model tiers ($3600$ outputs, thresholds fixed before any held-out cell was sent), inverting one documented mapping under a terse tool description costs $26.7$ points of exact terminal-state accuracy (95\% CI $[20.7, 33.0]$), and $73.1$ points on the omission collision. Writing the operation truth table into the description recovers $43.7$ points. Removing every worked example still recovers $39.8$ points. Explicit-operation schemas fall short of a pre-registered $8$-point bar on the one tier with room to improve. Three later pre-registered models ($2160$ outputs), the newest published by both vendors, show the same failure. We release the compiler, frozen manifests, $12{,}167$ deterministic audit checks and $7200$ scored traces.
Chat is not available.
Successful Page Load