GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions
Abstract
Understanding where and why Graphical User Interface (GUI) agents fail is essential for building more reliable systems, yet current evaluation relies on step accuracy, a metric that treats each screen independently and overlooks the underlying structure of GUI environments. This leads to two critical blind spots: (1) functionally equivalent screens are evaluated in isolation, obscuring systematic failure patterns across shared screens; and (2) the long-tailed GUI distribution renders failures on rare but critical screens invisible under standard metrics. To address these issues, we propose \textbf{GUITAR}, a state-centric diagnostic framework that performs structured failure analysis over both states and transitions, using a State Transition Graph (STG) by mapping visually diverse screens to shared functional states. Evaluated with 8 agents across 3 tasks on AndroidControl and 3 tasks on Mind2Web, GUITAR reveals that failures are both structurally concentrated and metrically invisible: 61.9\% of failure instances occur in 20\% states, consistently localizing to a small set of bottleneck states. Furthermore, leveraging bottleneck states for inference guidance yields +2.8\% performance improvement. These findings highlight structure-aware evaluation as necessary for GUI agent diagnosis, and demonstrate that targeted intervention on bottleneck states offers a practical path to improving reliability.