Fast calorimeter simulation is crucial for collider physics to accelerate the comparisons between theory and experiments. Physics simulators are often precise but slow to generate the required high granular detector response. Fast surrogate models based on machine learning models have shown great promise by leveraging modern computational hardware and their ability to capture, the complex and high dimensional space of calorimeter detectors. In this paper we introduce a new fast surrogate model based on latent diffusion models named CaloLatent, able to reproduce, with high fidelity, the detector response in a fraction of the time required by similar generative models. We evaluate the generation quality and speed using the Calorimeter Simulation Challenge 2022 dataset.