RAGBoost: Robust Tabular Learning via Retrieval-Augmented and Ancillary-Guided Gradient Boosting

Gradient Boosted Decision Trees (GBDTs) remain a leading approach for tabular learning, offering strong predictive performance, computational efficiency, and robustness to heterogeneous feature types. Yet their standard second-order leaf updates suffer from a node-level statistical limitation: as trees grow deeper, sparse leaves provide increasingly unreliable gradient estimates, while global regularization cannot adapt to local uncertainty or residual structure. We propose RAGBoost, a retrieval-augmented and ancillary-guided boosting framework for robust tabular learning. RAGBoost augments standard GBDT optimization with a two-layer node-adaptive correction mechanism. First, it retrieves historically observed leaves with similar gradient-state configurations to form node-conditional gradient priors, stabilizing variance-dominated sparse-leaf updates through empirical shrinkage. Second, it uses ancillary residual correction to detect and mitigate bias-dominated within-leaf residual structure that remains unresolved by the main tree. These corrections modify only leaf values, preserving the original tree structure and inference pipeline. Empirically, RAGBoost achieves the best overall average rank across 40 heterogeneous tabular benchmarks in our evaluation, while remaining strongly competitive with recent deep tabular and tabular foundation model baselines. On TabReD, an industrial-scale temporal-shift benchmark, RAGBoost outperforms XGBoost and the base TabM variant overall, suggesting improved robustness under realistic distribution drift.