Cross-Foundation Complementary Learning Systems for Continual Test-Time Adaptation in Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation (OVSS) models enable dense prediction for classes specified at test time, but continual test-time adaptation (CTTA) over long, non-stationary streams can corrupt their language-aligned semantic interface, which we diagnose through synonym-prompt text alignment. We propose Cross-Foundation Complementary Learning Systems (XF-CLS), a source-free OVSS-CTTA framework that makes lightweight online adaptation stable over long horizons. Inspired by Complementary Learning Systems, which couple rapid plastic learning with slow consolidation, XF-CLS decouples plasticity from stabilization: a single fast online adaptation path updates only a small set of visual normalization parameters, a horizon-aware, source-anchored slow parameter memory consolidates stable changes and recovers from drift, and a frozen cross-foundation structural observer supplies a non-drifting structural prior for reliability-aware recovery and dense output refinement. Rather than training an additional teacher or using cross-foundation pseudo-label supervision, the frozen path preserves the open-vocabulary interface while correcting degraded visual structure. On long-horizon Cityscapes-to-ACDC and OnDA streams, XF-CLS achieves 32.92 and 30.75 mIoU, improving the NACLIP no-adapt baseline by +9.00 and +4.13 mIoU, respectively, and outperforming continual OVSS-TTA baselines. Decomposition studies and text-alignment diagnostics show that source-anchored recovery prevents long-horizon collapse, while frozen cross-foundation refinement adds consistent gains without updating extra parameters. XF-CLS provides a stable, parameter-efficient route to source-free continual open-vocabulary dense prediction, with inference overhead dominated by one additional frozen-backbone forward pass.