Curriculum Learning for Safety Alignment

Direct Preference Optimization is a widely used approach for safety alignment, aiming to reduce harmful behaviors in large language models. However, prior work shows that it can be brittle and exhibits poor out-of-distribution (OOD) generalization \citep{qi2025safety}. To this end, in this paper we investigate whether curriculum learning can improve the robustness of DPO-based safety alignment. We propose \textbf{Staged-Competence}, a curriculum-based framework that organizes preference data by difficulty, employs competence-based sampling, and progressively updates the reference model during training. Averaged across three model families, Staged-Competence reduces OOD harmful response rates by 16\% and jailbreak attack success rates by 20\%, while preserving general capabilities and maintaining near-zero over-refusal. We further show that Staged-Competence (1) matches baseline safety with only 75\% of the training data, demonstrating improved data efficiency and (2) yields better separation between safe and unsafe responses. Staged-Competence is agnostic to the underlying policy optimization loss and can extend to other DPO variants and alignment domains other than safety. Our code and data can be found at: \url{link/upon/acceptance}.