AsyncOPD: How Stale Can On-Policy Distillation Be?

On-policy distillation (OPD) is an increasingly important way to post-train large language models (LLMs), but, like reinforcement learning (RL), it relies on rollouts from the model being optimized. For reasoning workloads, rollout generation can dominate training time. Asynchronous RL alleviates this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data; prior work studies how to stabilize learning from such data. However, it remains underexplored whether these asynchronous RL ideas and stale-data solutions transfer to OPD, and what OPD-specific constraints arise. To address this gap, we provide the first systematic study of staleness in asynchronous OPD. We first show that KL direction changes the stale-data problem: teacher-weighted forward KL is robust to stale rollouts, whereas student-weighted reverse KL is vulnerable. Second, for this vulnerable reverse-KL case, we study whether methods designed to stabilize asynchronous RL can mitigate OPD staleness. We find that they do not improve over a simpler OPD-specific surrogate: recomputing the reverse-KL signal under the current student at learner time without clipping. Third, we identify an OPD-specific teacher-cache constraint: under asynchronous execution, teacher scores are available at learner time only on cached actions. The resulting bias-variance tradeoff for sparse and sampled reverse-KL OPD implementations motivates multi-sample Monte Carlo (MC), which preserves MC correctability while reducing one-sample variance. Finally, we present and open-source **AsyncOPD**, a fully asynchronous OPD training pipeline built from these estimator choices. Experiments show that AsyncOPD improves training throughput by $1.6\times$ to $3.8\times$ over strict synchronous training while reaching comparable accuracy.