Early identification of changes to heart size is critical to improving outcomes in heart failure. We introduce a deep learning model for detecting cardiac structural abnormality in chest X-rays. State of the art deep learning models focus on detecting cardiomegaly, a label consistently shown to be a poor marker for cardiac disease. Our method targets four major cardiac structural abnormalities -- left ventricular hypertrophy (LVH), severe LVH, dilated cardiomyopathy phenotype, and hypertrophic cardiomyopathy phenotype -- with performance superior to radiologist assessments. Furthermore, upon interrogation, we find our model's predictions are driven most strongly by structural features of the heart, confirming our model correctly focuses on the elements of chest X-rays pertinent to the diagnoses of cardiac structural abnormality.