Protein Language Model Fitness is a Matter of Preference

Although protein language models (pLMs) have been used to successfully design proteins for therapeutic and research purposes, it remains unclear under what conditions they will succeed or fail. We show that pLM likelihoods indicate zero-shot fitness prediction capabilities. To determine what data causes a sequence to be likely, we utilize influence functions finding that homologous neighbors from proteins search are responsible for increasing sequence likelihood most. We use this to motivate finetuning on sequences with only low likelihoods to improve the performance on selecting beneficial mutations, thus improving protein engineering capabilities.