Incidental images change VLM decisions
Arnav Gowda ⋅ Charlotte Li ⋅ Sneheel Sarangi ⋅ Syed Islam
Abstract
Deployed vision-language models already refuse, give, bid, and judge disputes with extra pixels in context. Those pixels are usually not part of what caused the question. Visual-priming studies and Anthropic's text-only work on functional emotion concepts predict that an emotionally charged photograph should bias the answer. We tested that prediction on over-refusal, economic games, and a twelve-task open-ended board, using EMOTIC named emotions and OASIS valence tertiles as separate operationalizations. The photo's labeled emotion or valence almost never added an extra shift. Adding any photograph often did, including benign refusal rising from $0.125$ to $0.288$--$0.362$ on Gemma-4-E4B and from $0.000$ to $0.288$--$0.512$ on Gemma-4-12B. A reliable decision should ignore a picture that is not evidence. We see the opposite.
Chat is not available.
Successful Page Load