The application of reinforcement learning in traffic signal control (TSC) has been extensively researched and yielded notable achievements. Most of the existing works for TSC assume that all the traffic conditions surrounding the study intersections can be collected completely and continuously by the deployed sensors. However, due to the lack and error of sensors, data-missing scenarios often occur, leading to the assumption not to be held in practice. To meet the needs of practical applications, in this paper, we study the problem of TSC with missing data in the offline setting. To this end, we propose DiffLight, a conditional diffusion model, unifying the two tasks within this problem, i.e., traffic data imputation and decision-making tasks. Specifically, we introduce the partial rewards conditioned diffusion to avoid confusion about padded missing rewards. Meanwhile, the noise model is designed as a spatial-temporal transformer architecture to capture the spatial-temporal dependencies between intersections. Additionally, we propose the diffusion communication mechanism to enable communication and promote coordinated control among intersections. Extensive experiments on five datasets with different data-missing scenarios demonstrate that DiffLight is an effective controller to address TSC with missing data.