Increases in the accuracy and accessibility of machine learning, and particularly deep learning models, have driven their recent extensive use in a variety of fields, including digital histopathology. However, the impressive performance of these models for clinical tasks is counterbalanced by a lack of transparency – a significant obstacle for their usefulness in clinical contexts. Explainable AI (XAI) methods try to address this issue by providing explanations for model decisions. In image analysis, saliency maps are among the most popular techniques used. To date, no comprehensive study has been done of XAI methods evaluating different aspects (e.g., model fidelity, localization ability, and stability) utilizing different architectures. Understanding the relative efficacy of XAI methods via evaluation metrics, as well as the strengths and shortcomings of existing XAI approaches, will bolster translational work in this important field.