What DNA Foundation Models Learn Beyond Sequence Composition

A central question in genomic representation learning is whether DNA foundation models~(FMs) encode information beyond simple sequence composition, or whether their apparent representational power largely reflects signals already captured by classical compositional statistics. We address this by establishing $k$-mer frequency vectors as a rigorous biologically grounded baseline and introducing a geometric decomposition that partitions FM embeddings into a component linearly predictable from $k$-mer frequencies and an orthogonal residual representing genuinely non-compositional information. By evaluating each component separately as a downstream feature space, we directly measure what FMs encode beyond composition and whether that content is beneficial, neutral, or detrimental. Across 57 genomic classification datasets and 2 gene expression regression tasks, we compare five FMs (NTv3-650M, HyenaDNA, DNABERT-2, Caduceus-Ph, Evo2-1B) against $k$-mer features ($k \in \{4,5,6\}$) under a controlled frozen-embedding protocol with a fixed Random Forest classifier. NTv3 is the only model to consistently outperform $k$-mers (63.2\% of datasets, mean MCC $= 0.562$), with gains concentrated in splice-site and promoter tasks; HyenaDNA, DNABERT-2, and Evo2 are outperformed on 86\%, 95\%, and 93\% of datasets. Wilcoxon signed-rank tests with FDR correction confirm these patterns are systematic and robust to dataset heterogeneity. Our decomposition reveals that FM advantage originates exclusively from the non-compositional residual, and only in tasks with inherently positional regulatory signals. For HyenaDNA and DNABERT-2, the residual actively degrades classification. The fraction of embedding variance explained by $k$-mers ($R^2$) is uncorrelated with downstream quality, demonstrating that the quantity of non-compositional information is not a proxy for its utility. These results show that, in the frozen-embedding regime, current DNA FM representations largely recapitulate sequence composition, and provide a principled diagnostic for predicting which task conditions are likely to benefit from non-compositional structure.