Opening remarks and introduction to the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments.

As AIs are used in more common and consequential situations, it is important that we find ways to take advantage of our computational capabilities while also holding the creators of these systems accountable. In this talk, I'll start out by sharing some of the challenges associated with deploying AIs in healthcare, and how interpretability or explanation is an essential tool in this domain. Then I'll speak more broadly about the role of explanation in holding AIs accountable under the law (especially in the context of current regulation around AIs). In doing so, I hope to spark discussions about how we, as a machine learning community, believe that our work should be regulated.

The lack of interpretability remains a key barrier to the adoption of deep models in many healthcare applications. In this work, we explicitly regularize deep models so human users might step through the process behind their predictions in little time. Specifically, we train deep time-series models so their class-probability predictions have high accuracy while being closely modeled by decision trees with few nodes. On two clinical decision-making tasks, we demonstrate that this new tree-based regularization is distinct from simpler L2 or L1 penalties, resulting in more interpretable models without sacrificing predictive power.

We propose a method to optimise the parameters of a policy which will be used to safely perform a given task in a data-efficient manner. We train a Gaussian process model to capture the system dynamics, based on the PILCO framework. Our model has useful analytic properties, which allow closed form computation of error gradients and estimating the probability of violating given state space constraints. During training, as well as operation, only policies that are deemed safe are implemented on the real system, minimising the risk of failure.

[1] "Network Analysis for Explanation" [2] "Using prototypes to improve convolutional networks interpretability" [3] "Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning" [4] "Deep Reinforcement Learning for Sepsis Treatment" [5] "Analyzing Feature Relevance for Linear Reject Option SVM using Relevance Intervals" [6] "The Neural LASSO: Local Linear Sparsity for Interpretable Explanations" [7] "Detecting Bias in Black-Box Models Using Transparent Model Distillation" [8] "Data masking for privacy-sensitive learning" [9] "CLEAR-DR: Interpretable Computer Aided Diagnosis of Diabetic Retinopathy" [10] "Manipulating and Measuring Model Interpretability"

Machine learning systems, which are often used for high-stakes decisions, suffer from two mutually reinforcing problems: unfairness and opaqueness. Many popular models, though generally accurate, cannot express uncertainty about their predictions. Even in regimes where a model is inaccurate, users may trust the model's predictions too fully, and allow its biases to reinforce the user's own. In this work, we explore models that learn to defer. In our scheme, a model learns to classify accurately and fairly, but also to defer if necessary, passing judgment to a downstream decision-maker such as a human user. We further propose a learning algorithm which accounts for potential biases held by decision-makers later in a pipeline. Experiments on real-world datasets demonstrate that learning to defer can make a model not only more accurate but also less biased. Even when operated by biased users, we show that deferring models can still greatly improve the fairness of the entire pipeline.

Deep neural network (DNN) models have recently obtained state-of-the-art prediction accuracy for the transcription factor binding (TFBS) site classification task. However, it remains unclear how these approaches identify meaningful DNA sequence signals and give insights as to why TFs bind to certain locations. In this paper, we propose a toolkit called the Deep Motif Dashboard (DeMo Dashboard) which provides a suite of visualization strategies to extract motifs, or sequence patterns from deep neural network models for TFBS classification. We demonstrate how to visualize and understand three important DNN models using three visualization methods: saliency maps, temporal output scores, and class optimization. In addition to providing insights as to how each model makes its prediction, the visualization techniques indicate that CNN-RNN makes predictions by modeling both motifs as well as dependencies among them.

In this talk, we will formalize transparency as acting in a dynamical system or MDP in which we augment the physical state with the human's belief about the robot. We will characterize the dynamics model in this MDP, and show that approximate solutions lead to cars that drive in a way that is easier to anticipate, robots that come up with instructive demonstrations of their task knowledge, manipulator arms that clarify their intent, and navigation robots that clarify their future task plans. Lastly, we will briefly explore robots that express more interesting properties like the their level of confidence in their task, or the weight of an object they are carrying.