Recent advances in deep learning have led to the development of models approaching human level of accuracy. However, healthcare remains an area lacking in widespread adoption. The safety-critical nature of healthcare results in a natural reticence to put these black-box deep learning models into practice. In this paper, we explore interpretable methods for a clinical decision support system, sleep staging, based on physiological signals such as EEG, EOG, and EMG. A recent work has shown sleep staging using simple models and an exhaustive set of features can perform nearly as well as deep learning approaches but only for certain datasets. Moreover, the utility of these features from a clinical standpoint is unclear. On the other hand, the proposed framework, NormIntSleep shows that by representing deep learning embeddings using normalized features, great performance can be obtained across different datasets. NormIntSleep performs 4.5% better than the exhaustive feature-based approach and 1.5% better than other representation learning approaches. An empirical comparison between the utility of the interpretations of these models highlights the improved alignment with clinical expectations when performance is traded-off slightly.