Split and Aggregation Sensitivity in the ProteinGym Supervised Benchmark

Supervised models of protein fitness are typically evaluated by a single ProteinGym score computed over hundreds of deep mutational scanning assays. While this score answers the question of which model is best overall, it is often read as also answering two others: whether the rest of the ranking is equally established, and whether the same model is best for the protein of interest. We analyze the public ProteinGym v1.3 supervised-substitution tables, which cover 217 deep mutational scanning assays, 186 proteins, and 11 models, across the three cross-validation schemes, the three ways of combining assays into a single score, and both metrics that ProteinGym reports. Kermut is the best model in all 18 combinations, even though its score under ProteinGym's corrected average falls from 0.745 on the random split to 0.593 on the contiguous split. The rest of the ranking is not established: with equal protein weighting, 11 of 55 model pairs switch position between the random and contiguous splits, and those two splits share a best-scoring model for only 77 of 186 proteins. In the absence of further information, choosing the overall winner is warranted, since Kermut is still best for 107 of the 186 proteins under the contiguous split. The best model for a particular protein nevertheless depends on the evaluation choice, so a benchmark average should be reported together with the weighting rule behind it and the fraction of proteins whose best model changes.