In recent years, there has been a proliferation of ready-to-use large language models (LLMs) designed for various applications, both general-purpose and domain-specific. Instead of advocating for the development of a new model or continuous pretraining of an existing one, this paper introduces a pragmatic teacher-teacher framework to facilitate mutual learning between two pre-existing models.By leveraging two teacher models possessing complementary knowledge, we introduce a LIghtweight kNowledge alignmEnt (LINE) module aimed at harmonizing their knowledge within a unified representation space. This framework is especially useful in clinical settings where stringent regulations govern the utilization of real-life clinical notes. Specifically speaking, our trained LINE module adeptly captures essential information from clinical notes using highly de-identified data. Validation and downstream tasks are conducted to showcase the efficacy of the proposed framework.