Large pretrained foundation models exhibit outstanding performance or even surpass human experts in some high-stakes applications. However, most of these models are currently evaluated primarily on prediction accuracy, overlooking the validity of the rationales behind their accurate predictions. There exists a pressing need to ensure the dual-correctness of predictions, i.e., correct prediction for correct rationales, for the safe deployment of foundation models. To this end, we propose a two-phase scheme for developing dual-correct predictions. We first curate a new dataset by representing prediction rationale in a machine-readable format, then leverage the relations between rationale to guide the learning of models without manual annotations. The extensive experiments and ablation studies show that our model outperforms the state-of-the-art and fine-tuned models in a wide range of tasks including classification and retrieval. Furthermore, our method significantly improves the model's rationale correctness in terms of rationale localization and disentanglement.