Existing approximate inference techniques produce predictive distributions that are quite distinct from the predictive distribution of the gold-standard Hamiltonian Monte Carlo. In this work, we bring the predictive distribution produced by deep ensembles more closer to the Hamiltonian Monte Carlo predictive distribution by increasing the diversity within the ensembles. The proposed approach outperforms the existing approximate inference methods and is also currently ranked the highest in the Approximate Inference competition at NeurIPS 2021.