Deep neural networks have demonstrated remarkable performance in various vision tasks, yet their performance heavily rely on the quality of training data. Noisy label is a critical issue in medical dataset and can significantly degrade the model performance. Previous clean sample selection methods do not utilize the well pre-trained feature of vision foundation model (VFM) and assumed that training is start from scratch. In this paper, we propose Cufit, a curriculum fine-tuning paradigm of VFM for medical image classification under label noise. Our method is motivated by the fact that the linear probing of VFM relatively free from the noisy samples since it does not update feature extractor VFM, thus robustly classify the training samples. Subsequently, curriculum fine-tuning of two adapters is conducted by clean sample selection started from the linear probing. Our experimental results demonstrate that Cufit outperforms previous methods in various medical image benchmarks. Specifically, our method surpasses previous baselines 5.0\%, 2.1\%, 4.6\%, and 5.8\% on a 40\% noise rate of HAM10000, APTOS-2019, BloodMnist, and OrgancMnist, respectively. Furthermore, we provide extensive analyses to demonstrate impact of the our method on noisy label detection. For example, our method shows high label precision and label recall compared to the previous method. Our code will be available upon publication.