Meaning Through Motion: DUET – A Multimodal Dataset for Kinesics Analysis in Dyadic Activities

Human activity recognition (HAR) has advanced significantly with the availability of diverse datasets, yet the field remains constrained by the scarcity of resources focusing on contextualizable two-person, or “dyadic” interactions. Existing datasets primarily aim to help improve the recognition of physical coordination in single-person settings, overlooking the intricate dynamics and kinesics present in interactions between two individuals. To address this gap, we introduce the Dyadic User EngagemenT (DUET), a comprehensive dataset designed to enhance the understanding and recognition of interactions. DUET comprises 12 interactions adopted from a taxonomy rooted in psychology to distill social semantics embedded in bodily movements. The marriage of HAR and social context dependencies sets the stage for refining recognition accuracy, improving the authenticity of telepresence avatar, automating sociological and psychological examinations, and many more. To support applications with different purposes and constraints, every sample spans across four modalities: RGB, depth, infrared, and 3D skeleton joints. Besides, the dataset was collected at three locations, including an open indoor space, a confined indoor space, and an open outdoor space. The variety of data collection locations helps improve the resilience against background variation and investigate the effect ambient environment imposes on HAR algorithms. In total, we collected 14,400 samples utilizing a novel technique that captures interactions from multiple views with a single camera. The technique diversifies how interactions are observed and yields the highest sample-class ratio known to date. We benchmark six state-of-the-art HAR algorithms on DUET, demonstrating the dataset’s complexity and current model’s limitations in recognizing dyadic interactions. DUET is publicly available at https://huggingface.co/datasets/saluslab/DUET, providing a valuable resource for the research community to advance HAR in dyadic settings.