Timezone: »

Learning Disentangled Behavior Embeddings
Changhao Shi · Sivan Schwartz · Shahar Levy · Shay Achvat · Maisan Abboud · Amir Ghanayim · Jackie Schiller · Gal Mishne

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @

To understand the relationship between behavior and neural activity, experiments in neuroscience often include an animal performing a repeated behavior such as a motor task. Recent progress in computer vision and deep learning has shown great potential in the automated analysis of behavior by leveraging large and high-quality video datasets. In this paper, we design Disentangled Behavior Embedding (DBE) to learn robust behavioral embeddings from unlabeled, multi-view, high-resolution behavioral videos across different animals and multiple sessions. We further combine DBE with a stochastic temporal model to propose Variational Disentangled Behavior Embedding (VDBE), an end-to-end approach that learns meaningful discrete behavior representations and generates interpretable behavioral videos. Our models learn consistent behavior representations by explicitly disentangling the dynamic behavioral factors (pose) from time-invariant, non-behavioral nuisance factors (context) in a deep autoencoder, and exploit the temporal structures of pose dynamics. Compared to competing approaches, DBE and VDBE enjoy superior performance on downstream tasks such as fine-grained behavioral motif generation and behavior decoding.

Author Information

Changhao Shi (UC San Diego)
Sivan Schwartz (Technion - Israel Institute of Technology)
Shahar Levy (Technion - Israel Institute of Technology)
Shay Achvat (Technion, Technion)
Maisan Abboud (Technion - Israel Institute of Technology)
Amir Ghanayim (Technion - Israel Institute of Technology)
Jackie Schiller (Technion - Israel Institute of Technology)
Gal Mishne (UC San Diego)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors