Critical Mass: A Fixed Threshold is Not a Fixed Benchmark
Tomo Oga ⋅ Devesh Shah ⋅ Cailum M Stienstra ⋅ Gabriel Asher ⋅ Antonio Henrique de Oliveira Fonseca ⋅ Niall O'Connor ⋅ Michael Widrich
Abstract
Molecular machine learning evaluates out-of-distribution performance with structure disjoint splits: threshold a pairwise distance, then assign whole connected components to folds. For predicting structures from mass spectra, the field has converged on single-linkage clustering under the myopic Maximum Common Edge Subgraph (MCES) distance at a threshold of 10, a convention set on 29,000 molecules. Because the criterion is fixed, a larger library should buy a larger and more chemically diverse held-out set under the same guarantee. Public libraries now hold 227,690 molecules, and the rule has never been checked at that scale: MCES is NP-hard per pair over a quadratically growing set. We make the check feasible with admissible bounds and a union-find skip that reduces $2.6 \times 10^{10}$ pairs to $1.15 \times 10^{7}$ integer programs, a $2{,}259\times$ saving, while provably producing the identical partition. At this scale the graph percolates. A single component holds $95.5$% of the library, capping any test set at $4.5$%. What remains is not merely smaller, but systematically different: a held-out compound is heavier than a training compound $88.1$% of the time. Ten edits is a large fraction of a small molecular graph and a small fraction of a large one, so heavy molecules are the ones that fail to connect. Large library benchmarks using this splitting rule naively therefore conflate structural novelty with extrapolation to larger, more complex chemistry.
Chat is not available.
Successful Page Load