In learning with noisy labels, the noise transition matrix reveals how an instance relates from its clean label to its noisy label. Accurately estimating an instance's noise transition matrix is crucial for inferring its clean label. However, when only a noisy dataset is available, noise transition matrices can be estimated only for some "special" instances. To leverage these estimated transition matrices to help estimate transition matrices of other instances, it is essential to explore relations between the matrices of these "special" instances and those of the others.Existing studies usually build the relation by explicitly defining the similarity between the estimated noise transition matrices of "special" instances and those of other instances. However, these similarity-based assumptions are hard to validate and may not be aligned with real-world data. If these assumptions fail, noise transition matrices and clean labels cannot be accurately estimated.In this paper, we found that by learning the latent causal structure governing the generative process of noisy data, we can estimate noise transition matrices directly, eliminating the need for similarity-based assumptions. To achieve this, unlike previous generative label-noise learning methods, we consider causal influences between latent causal variables and model them with a learnable graphical model. Utilizing only noisy data, our method can effectively learn the latent causal structure. Experimental results on various label-noise datasets demonstrate that our approach achieves state-of-the-art performance in estimating noise transition matrices, which leads to the improvement of classification accuracy.