Neighborhood environments play a significant role in shaping the well-being of individuals and communities, consequently contributing to inequality in the United States. Drawing on a dataset of street view images of buildings in five U.S. cities collected over 13 years, we train a convolutional neural network to estimate the amount of deterioration in the building shown in each image. We then use the trained model to detect trends in the level of building blight in Boston from 2007 to 2017. Our results show that the changes in upkeep correspond to overall economic trends, specifically the Great Recession and the subsequent recovery.