Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations

Concept Bottleneck Models (CBMs) are widely promoted as a pathway to interpretable and potentially more robust learning, yet existing evidence on their robustness remains mixed and often contradictory. We argue that these discrepancies arise from conflating different robustness notions and perturbation regimes, rather than from fundamental disagreements about CBMs themselves. To disentangle these factors, we introduce a generator-based evaluation framework that enables controlled comparisons between standard classifiers and CBMs under two distinct perturbation types: continuous geometric perturbations in latent space and discrete semantic interventions in concept space. Within this framework, we evaluate robustness both empirically, via prediction and concept-level sensitivity metrics, and certifiably, using randomized smoothing in latent and concept spaces. Across experiments on CUB‑200‑2011 and RIVAL‑10 variants, we reconcile previously conflicting findings by clarifying when, and in what sense, concept bottlenecks do or do not improve robustness. By further analyzing robustness under varying task conditions, including class semantic similarity and concept vocabulary size, we show that interpretability does not inherently confer robustness. Instead, concept bottlenecks shift where and how sensitivity manifests, revealing a nuanced interpretability–robustness trade‑off that depends critically on the perturbation regime and task structure. Together, the framework and findings clarify how concept bottlenecks influence robustness and provide guidance for designing CBMs that better balance interpretability and robustness.