From Synthetic Databases to Relational Representations: Role of Curricula, DFS, and Depth for Graph-Based Foundation Models

Prior-data fitted networks (PFNs) transfer from synthetic pretraining to unseen datasets without fine-tuning. A prior extended abstract carried this recipe to graph-based relational foundation models, pretraining the message-passing encoder Griffin on synthetic relational databases and reading it out with a frozen in-context TabICL head; an ordered curriculum from schema-guided to schema-agnostic data gave the strongest frozen encoders on 16 real-world classification tasks. Here we test the limits of that recipe. Keeping every classification result unchanged, we add eight regression tasks, hop-bounded Deep Feature Synthesis (DFS) aggregates, 4-hop subgraphs, and fine-tuning the encoder into the head. Frozen synthetic pretraining proves effective mainly for classification: on the full regression suite no frozen encoder beats raw target-row features, and curriculum order, while it still matters, barely moves regression. Hop-bounded DFS aggregates improve both metrics as hop depth increases and are the only frozen intervention to outperform raw features on regression. Deeper subgraphs lie dormant when frozen; fine-tuning unlocks them and yields the study's best classification and regression scores.