Convolutional neural networks (CNN) are widely used in medical images diagnostic. However, training the CNNs is prohibitive in a low-data environment. In this study, for the low-data medical image domain, we propose a novel knowledge transfer approach to facilitate the training of CNNs. Our approach adopts the attention transfer framework to transfer knowledge from a carefully pre-trained CNN teacher to a student CNN. The performance of the CNN models is then evaluated on three medical image datasets including Diabetic Retinopathy, CheXpert, and ChestX-ray8. We compare our results with the well-known and widely used transfer learning approach. We show that the teacher-student (Attention transfer) framework not only outperforms transfer learning, in both in-domain and cross-domain knowledge transfer but also behave as a regularizer.