Exploring Multi-Order Self-Similarity for Motion Understanding

Space-time self-similarity (STSS), which captures visual correspondences across frames, provides an effective way to represent temporal dynamics for video understanding.In this work, we explore higher-order STSS and demonstrate how STSSs at different orders reveal distinct aspects of these dynamics. We then introduce the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module designed to learn and integrate multi-order STSS features. It can be applied to a wide range of motion-centric tasks with only marginal computational cost and memory overhead. Extensive experiments on video action recognition, motion-centric video VQA, and robotic tasks consistently demonstrate substantial improvements, validating the broad applicability of MOSS as a general motion modeling module. The source code and checkpoints will be publicly available.