When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation

Recent segmentation-based trackers built on foundation models such as Segment Anything Model (SAM) achieve strong performance through memory-based mask propagation. However, despite their strong generalization ability, these methods remain brittle under occlusion, reappearance, and distractor scenarios, where errors accumulate and lead to identity drift. Existing approaches attempt to improve robustness through better memory design and temporal modeling, yet they implicitly assume that propagated predictions remain reliable over time, lacking a mechanism to identify and correct failures. To address this limitation, we propose a failure-aware tracking framework that augments segmentation-based propagation with explicit reasoning and selective recovery. Our key idea is to decouple tracking into two complementary processes: verifying prediction reliability and recovering the target when failures occur. Specifically, we introduce a dual-agent architecture consisting of a vision-language verification agent and a correction agent. The verification agent reasons over spatio-temporal context to assess whether current predictions remain consistent with the target object, enabling the detection of identity switches, missed recoveries, and distractor-induced failures. When unreliable predictions are identified, the correction agent is selectively activated to recover the target using temporal cues. To further improve robustness under ambiguous scenarios, we introduce targeted training perturbations that simulate identity switches and and distractor-induced drift during training. By explicitly modeling failure detection and recovery within tracking loop, our framework transforms tracking from a purely propagation-based process into a self-correcting system. Experimental results on LVOS-v2 and SecVOS show that the proposed framework improves SAM3 by 1.2 J&F and 2.7 J&F respectively, demonstrating the value of explicit reasoning and selective recovery in segmentation-based tracking.