Corrective Diffusion Language Models

While Diffusion Language Models (DLMs) are theoretically well-suited for iterative refinement due to their non-causal structure, off-the-shelf masked diffusion language models (MDLMs) often fail to reliably revise incorrect tokens in practice. The key challenge lies in the model's inability to distinguish between correct and erroneous tokens in a visible sequence. Standard MDLM training is restricted to the objective of unmasking, undermining the effectiveness of refinement guided by confidence. Based on this observation, we study corrective behavior in DLMs, defined as the ability to assign lower confidence to incorrect tokens and iteratively refine them while preserving correct content. Under matched compute and data, we show that this capability is not induced by conventional masked diffusion objectives and propose a post-training principle oriented by correction that explicitly supervises visible incorrect tokens, enabling discriminative confidence and targeted refinement. To evaluate corrective behavior, we introduce the Code Revision Benchmark, a controllable and executable benchmark for assessing error localization and in-place correction. Beyond improving in-place correction, our post-training principle yields stronger general generation as a direct consequence: the same error-aware confidence that drives reliable revision also makes parallel decoding more effective and improves answer quality on standard generation tasks. We validate this dual benefit across three distinct domains, code completion, mathematical reasoning, and high-entropy parallel decoding (ParallelBench), where models trained with our objective consistently outperform both the base model and standard masked-diffusion fine-tuning.