$\boldsymbol{f}$-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control

Scaling on-policy distillation (OPD) for large language models (LLMs) confronts a fundamental tension: asynchronous execution is necessary for system efficiency but structurally deviates from the ideal on-policy objective. To address this challenge, we theoretically decompose the objective discrepancy into rollout drift and supervision drift, capturing staleness in student occupancy and teacher context, respectively. Building on this, we introduce a sample-level freshness score that quantifies the reliability of buffered sample with respect to the on-policy objective. Guided by this signal, we further propose $\boldsymbol{f}$**-OPD**, a novel framework that adaptively regulates stale-sample influence and constrains policy drift accumulated under asynchronous optimization. Across reasoning, tool-use, and coding-agent tasks of increasing interaction horizon, $f$-OPD consistently achieves task performance comparable to synchronous training while retaining the throughput advantages of asynchronous execution. Our results establish the first recipe that seeks to achieve a performance–efficiency trade-off in OPD, paving the way for long-horizon agentic post-training at scale.