What a Norm-Matched Control Does Not Bound: A Difference-of-Means Direction in a Vision-Language Model That Passes the Standard Bar

Current mechanistic interpretability methods use difference-of-means and activation steering to identify internal representations in Vision Language Models (VLMs) to understand why the model behaves the way it does. Successful activation steering does not guarantee that the extracted direction for a visual attribute actually represents the intended attribute. Contrastive datasets can confound the target attribute with answer selection variables. We show that difference-of-means extraction for contrasting prompts can extract an answer choice component rather than the intended visual attribute using two controlled multi-image attribute binding experiments. Our first experiment shows that the queried color and the correct answer choice are equally distributed across the dataset. However, steering the extracted direction fitted from the color contrasts resulted in a decay to a single answer letter that yielded $50$\% accuracy rather than $0$\% which would be expected for an intervention that exchanges the representation for two images in the model. Our second experiment uses a legend swapping mechanism which keeps the images and prompt identical but swaps the answer letter choices for two images, which allows us to distinguish visual slot binding from answer token effects. The same extracted direction fitted for the color attribute also exhibits a similar behavior when applied to a shape question, despite being extracted exclusively for the color attribute, and it is well represented by the answer direction. Together, these two experiments show that activation steering is insufficient to establish that an extracted direction represents the intended attribute. Our findings suggest that stronger counterbalancing and specificity controls should be implemented when interpreting contrastive directions in VLMs.