Ranking-based loss functions have recently been shown to improve the quality of predictions of fitness landscapes for both standard supervised deep learning models and fine-tuned protein language models. We consider the implications of this finding for protein design with Bayesian optimisation. We investigate uncertainty quantification techniques applicable to protein language models fine-tuned with ranking losses, and show that they offer competitive calibration to CNN ensembles while demonstrating superior predictive performance. Finally, we demonstrate how uncertainty-aware ranking-based models can be exploited for protein design within the framework of preferential Bayesian optimisation.