Timezone: »

 
Poster
Compositional Explanations of Neurons
Jesse Mu · Jacob Andreas

Thu Dec 10 09:00 AM -- 11:00 AM (PST) @ Poster Session 5 #1592
in Poster Session 6 »

We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts that closely approximate neuron behavior. Compared to prior work that uses atomic labels as explanations, analyzing neurons compositionally allows us to more precisely and expressively characterize their behavior. We use this procedure to answer several questions on interpretability in models for vision and natural language processing. First, we examine the kinds of abstractions learned by neurons. In image classification, we find that many neurons learn highly abstract but semantically coherent visual concepts, while other polysemantic neurons detect multiple unrelated features; in natural language inference (NLI), neurons learn shallow lexical heuristics from dataset biases. Second, we see whether compositional explanations give us insight into model performance: vision neurons that detect human-interpretable concepts are positively correlated with task performance, while NLI neurons that fire for shallow heuristics are negatively correlated with task performance. Finally, we show how compositional explanations provide an accessible way for end users to produce simple "copy-paste" adversarial examples that change model behavior in predictable ways.

Author Information

Jesse Mu (Stanford University)
Jacob Andreas (MIT)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors