CausalAffect: Causally Guided Learning of Psychology-Aligned Facial Affect Relations

Facial affect understanding is not merely an image-to-label prediction problem: it requires identifying activated facial Action Units (AUs), modeling how AUs facilitate or inhibit one another, and explaining how AU configurations give rise to expression states. Such causal relations have long been studied in cognitive psychology and psychophysiology, where FACS, the Component Process Model, and motor synergy theories provide rich causal priors about facial behavior. In computer vision, however, data-driven facial affect models still largely learn correlational structures; even structured AU-aware methods often produce dependencies that are not well aligned with psychological priors. This leaves a persistent gap between biological facial mechanisms and learned visual representations, as observational facial images do not provide direct biological interventions and existing methods lack a principled way to recover psychology-aligned relations from data. We propose CausalAffect, a weakly supervised, causally guided framework for learning psychologically aligned facial affect relations from data. CausalAffect models two complementary relation types, AU$\rightarrow$AU and AU$\rightarrow$Expression, within a two-level hierarchy: a global graph capturing population-level psychology-aligned relations, and a sample-adaptive graph refining this backbone for individual variability. Reliable relation learning is supported by disentangled AU bottleneck representations, polarity-aware message passing, and feature-level counterfactual intervention. The learned relations recover canonical literature-supported pathways, reveal plausible inhibitory and sample-specific dependencies, are positively validated in a blind expert study, and exhibit intervention-consistent behavior under image-level AU edits. Experiments on six benchmarks show consistent improvements on both AU detection and facial expression recognition.