`

Timezone: »

 
Poster
TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
DONGXU LI · Chenchen Xu · Xin Yu · Kaihao Zhang · Benjamin Swift · Hanna Suominen · Hongdong Li

Wed Dec 09 09:00 PM -- 11:00 PM (PST) @ Poster Session 4 #1148

Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Sign videos consist of continuous sequences of sign gestures with no clear boundaries in between. Existing SLT models usually represent sign visual features in a frame-wise manner so as to avoid needing to explicitly segmenting the videos into isolated signs. However, these methods neglect the temporal information of signs and lead to substantial ambiguity in translation. In this paper, we explore the temporal semantic structures of sign videos to learn more discriminative features. To this end, we first present a novel sign video segment representation which takes into account multiple temporal granularities, thus alleviating the need for accurate video segmentation. Taking advantage of the proposed segment representation, we develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet. Specifically, TSPNet introduces an inter-scale attention to evaluate and enhance local semantic consistency of sign segments and an intra-scale attention to resolve semantic ambiguity by using non-local video context. Experiments show that our TSPNet outperforms the state-of-the-art with significant improvements on the BLEU score (from 9.58 to 13.41) and ROUGE score (from 31.80 to 34.96) on the largest commonly used SLT dataset. Our implementation is available at https://github.com/verashira/TSPNet.

Author Information

DONGXU LI (THE AUSTRALIAN NATIONAL UNIVERSITY)
Chenchen Xu (The Australian National University)
Xin Yu (University of Technology Sydney)
Kaihao Zhang (Australian National University)
Benjamin Swift (Australian National University)
Hanna Suominen (The Australian National University and Data61/CSIRO)
Hongdong Li (Australian National University)

More from the Same Authors