From Vision to Action: Enabling Real-World Agentic VLMs

Mobile and desktop task automation using agentic vision-language models (VLMs) faces critical safety challenges: processing sensitive UI screenshots containing personally identifiable information (PII), generating potentially harmful GUI actions, and operating autonomously without human oversight. Current approaches like VisionTasker rely on post-execution validation or lack generation-time safety guarantees entirely. We present SafeAgent-MCP, a framework combining semantic-constrained decoding with context-aware entity detection for generation-time safety and Model Context Protocol (MCP) for dynamic policy management. Our system enforces three constraint types: (1) Entity-level blocking using context- aware semantic entity recognition to prevent PII leakage from screenshots, (2) Action-space constraints via NVIDIA NIM structured generation restricting dangerous GUI operations, and (3) Policy-aware refusal leveraging OpenAI’s gpt-oss- safeguard for reasoning-based safety validation. SafeAgent-MCP provides the first systematic safety framework for production GUI automation agents combining modular semantic entity recognition with industry-standard constrained generation infrastructure.