Despite alleviating the dependence on dense annotations inherent to fully supervised methods, weakly supervised point cloud semantic segmentation suffers from the inadequacy of supervision signals due to annotation sparsity. In response to this challenge, we introduce a novel perspective that imparts supplementary supervision signals by regulating the feature space under weak supervision. Our initial investigation identifies which distributions accurately characterize the feature space in fully supervised learning, subsequently leveraging this priori to guide the alignment of the weakly supervised embeddings. Specifically, we explore two aspects, distance metrics and distribution modeling, and discover that the mixture of von Mises-Fisher distributions (moVMF) with cosine similarity excels in both fitting accuracy and generalizability. Accordingly, we develop a Distribution Guidance Network (DGNet), which comprises a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined feature space. Extensive experiments validate the rationality and effectiveness of our distribution choice and network design. Consequently, DGNet achieves state-of-the-art performance under multiple datasets and various weakly supervised settings. Additionally, DGNet provides probabilistic explanations for predictions from the perspective of the Bayesian theorem.