On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models

In this paper, we identify and investigate a critical issue in ODE-based sampling for diffusion models, referred to as the "collapse error," where samples tend to collapse locally. This phenomenon is observed across various datasets, including the MNIST, Mixture of Gaussians (MoG), and other synthetic datasets. Our analysis shows that this error occurs even in the early sampling process, with the error progressively accumulating and amplifying throughout the entire ODE sampling. To better understand the collapse error, we explore several factors that influence the collapse errors, including data distribution, model size, and training settings. Furthermore, We apply a set of techniques to mitigate the collapse error: (1) using an SDE-based sampler, (2) training different models for different segments of the diffusion process, and (3) employing new parameterizations for models. We hope this paper will draw attention to the "collapse error" phenomenon and encourage further research to better understand and address this issue in diffusion models.