As AIs are used in more common and consequential situations, it is important that we find ways to take advantage of our computational capabilities while also holding the creators of these systems accountable. In this talk, I'll start out by sharing some of the challenges associated with deploying AIs in healthcare, and how interpretability or explanation is an essential tool in this domain. Then I'll speak more broadly about the role of explanation in holding AIs accountable under the law (especially in the context of current regulation around AIs). In doing so, I hope to spark discussions about how we, as a machine learning community, believe that our work should be regulated.