Enhancing Contextual Privacy in VLM Geolocation Disclosure via Inference-Time Steering

Vision-language models (VLMs) can infer sensitive information such as geolocation from subtle visual cues. However, the appropriate level of disclosure depends on context. We find that contextually appropriate disclosure levels are substantially more recoverable from VLMs’ latent context than is reflected in their generated responses, revealing a gap between internal representation and realized disclosure behavior. Motivated by this, we introduce a lightweight inference-time approach for contextual disclosure control in frozen VLMs. Our method projects image-query representations onto an ordinal disclosure axis to infer privacy granularity a context permits, and then steers generation toward the corresponding behavioral prototype. Experiments on VLM-GeoPrivacy with different VLM backbones show that our approach improves contextual privacy alignment via better balancing over- and under-disclosure. These results demonstrate that inference-time steering can translate latent contextual signals into privacy-aligned disclosure without updating model parameters.