Studying protein mutations within amino acid sequences holds tremendous significance in life sciences. Protein language models (PLMs) have demonstrated strong capabilities in broad biological applications. However, they fall short in explaining and engineering protein mutations, due to their architectural design and lack of supervision. In this work, we present MutaPLM, a unified framework for interpreting and navigating protein mutations with protein language models. MutaPLM involves a protein delta network that captures explicit protein mutation representations within a unified feature space, and a transfer learning pipeline with a chain-of-thought (CoT) strategy to harvest protein mutation knowledge from biomedical texts. We also construct MutaDescribe, a large-scale protein mutation dataset with rich textual annotations, to provide cross-modal supervision signals. Through comprehensive experiments, we demonstrate that MutaPLM excels at providing human-understandable explanations for mutational effects and prioritizing novel mutations with desirable properties. Our code, model, and data will be publicly available.