A trustworthy AI system demands the ability to learn a broad range of knowledge with modest data and transfer the learned prior to the concrete task. In this work, we shall discuss the unsupervised meta-learning. We propose to learn multi-modal task-specific priors in the latent space using energy-based prior model, where the energy term couples a continuous latent vector and a symbolic one-hot label. Such coupling in the latent space informs the latent vector of the underlying category from the observed example. Our model can be learned in an unsupervised manner in the meta-training phase and evaluated in a semi-supervised manner in the meta-test phase. Our experiments show our method outperforms all the state-of-the-arts on miniImageNet and gives competitive results on Omniglot.