When a Window Is Not an Action: Selective Phase-Script Deliberation for Sliding-Window Human Activity Recognition

Sliding-window human activity recognition (HAR) splits continuous sensor streams into fixed-length, overlapping windows and trains a classifier to assign one activity label to each window. Window length and stride are chosen from sampling rate, latency or computation constraints, and assumptions about typical motion duration, but they need not align with motion cycles or activity changes. Some windows therefore contain only part of a repeated motion, fragments shared by related activities, or mixed evidence near a transition. Yet each window still receives a single label, creating a granularity mismatch between activity-level supervision and local motion evidence. We propose PSDNet (Phase-Script Deliberation Network), which separates local phase interpretation from final activity classification. PSDNet encodes the current window into latent phase primitives, combines them with recent history, and produces an initial classification with an uncertainty score. When uncertainty remains high, a deliberation module compares the current phase evidence with learned class-specific phase scripts, i.e., compact templates of short-term phase patterns, and updates classification scores over a small candidate set. An auxiliary boundary objective encourages sensitivity to possible activity changes without serving as a hard routing rule. This design concentrates extra computation where direct classification is least reliable while keeping easy cases efficient. Experiments on eight public HAR benchmarks show that PSDNet consistently improves accuracy and weighted F1 over strong CNN- and sequence-based baselines. Additional analyses on ambiguous windows and similar-activity confusion pairs support phase-aware selective deliberation for windows whose local evidence is insufficient for reliable classification.