Reference-Guided Training: Adaptive Gradient Scaling via Per-Sample Loss Comparisons

Empirical Risk Minimization treats training samples uniformly, which can degrade performance in noisy or heterogeneous settings. We introduce a reference-guided objective that uses per-sample error comparisons with a fixed reference model to reweight gradient contributions without relying on prediction imitation, while preserving gradient direction and ensuring bounded scaling. Theoretical analysis shows that the formulation induces adaptive weighting with controlled amplification. Experiments on image classification with synthetic label noise and on time-series forecasting demonstrate broad improvements across architectures, particularly in moderate-to-high noise and heterogeneous regimes. Results further indicate that the method is most effective when the error-ranking alignment between the baseline model and the reference model is low, suggesting that gains arise primarily from the diversity of the reference error ranking relative to the learner, rather than the reference's absolute performance.