torchgeo-bench: An Evaluation Harness for Geospatial Foundation Models

Geospatial foundation models (GFMs) have been a dominant research focus in the Earth observation community. However, the community lacks infrastructure for cheap, reproducible, apples-to-apples model comparisons under a fixed evaluation protocol. Each paper chooses different datasets, input processing, and probe recipes, making reported scores difficult to interpret or reproduce. We present torchgeo-bench, an open-source evaluation harness, leaderboard, and results repository for geospatial foundation models. It performs fast k-NN and linear probes of frozen models on common geospatial machine learning datasets, makes sensor and band handling explicit, and records the complete protocol with each result. Benchmarking a new model with torchgeo-bench requires only a small wrapper class that declares its expected inputs and returns embeddings. torchgeo-bench then handles preprocessing, probing, uncertainty estimates, and result logging. This makes controlled evaluations inexpensive and easy to reproduce and extend, providing a common evaluation layer for the GFM ecosystem. Code, configurations, and reference results are available at https://anonymous.4open.science/r/torchgeo-bench-71C8/.