Forced Choice Is Not a Tolerance What Multiple-Choice Grading Accepts on BixBench's Numeric Questions

BixBench evaluates AI agents on biological data analysis through free-text responses. One of its scoring methods uses a second model to map each response to one of four multiple-choice options. For numeric questions, a simpler alternative is to accept a response if it falls within (5\%) of the correct value (the key). We compare these grading methods using BixBench’s published runs, new runs of open-weight models, and runs of three current agents. The forced-choice grader accepts (17\%)–(25\%) of incorrect answers, a rate comparable to random guessing. These errors are not random, however: in the published GPT-4o and Claude 3.5 Sonnet runs, incorrect answers are accepted (3.5) and (6.0) times as often, respectively, when the key is the option nearest to the answer. Scores therefore depend on the placement of the options, with the three current agents scoring (10)–(12) percentage points higher under forced-choice grading than under the (5\%) tolerance check. The standard correction for guessing does not resolve this dependence. Option construction also exposes information about the key: in BixBench v1.5, the key is the second-smallest option in (51\%) of numeric questions. Randomizing the key’s rank removes this positional cue but introduces another failure mode: when the key is an extreme option, a nearest-option grader accepts every incorrect answer beyond it. These findings support grading numeric responses directly against the key using an explicit tolerance rather than forced-choice matching.