Timezone: »

 
NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation
David Alfonso-Hermelo · Ahmad Rashid · Abbas Ghaddar · Philippe Langlais · Mehdi Rezagholizadeh

@
in Dataset and Benchmark Poster Session 1 »

Slot-filling and intent detection are the backbone of conversational agents such as voice assistants, and are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios is yet to be demonstrated. In this work, we present NATURE, a set of simple spoken-language-oriented transformations, applied to the evaluation set of datasets, to introduce human spoken language variations while preserving the semantics of an utterance. We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly. Through our experiments we demonstrate that when NATURE operators are applied to evaluation set of popular benchmarks the model accuracy can drop by up to 40%.

[ Chat] 

Author Information

David Alfonso-Hermelo (Huawei Technologies Ltd.)

David Alfonso-Hermelo is an associate researcher at the Huawei Noah's Ark Lab in Montreal. Prior to joining Huawei, he worked at the RALI lab of the University of Montreal as research agent. He has obtained 3 MSc degrees: in Natural Language Processing from the Sorbonne Nouvelle University, in Language Sciences from Grenoble III University and Applied Linguistics from the University of Havana. His current research interests are Natural Language Processing, Knowledge Distillation, semantics representation for neural models and user-computer communication.

Ahmad Rashid (Huawei Technologies)
Abbas Ghaddar (Huawei Noah's Ark Lab, Montreal Research Center, Canada)
Philippe Langlais (University of Montreal)
Mehdi Rezagholizadeh (Huawei Technologies)

More from the Same Authors