Timezone: »
This tutorial will describe recent advances in deep learning techniques for Natural Language Processing (NLP). Traditional NLP approaches favour shallow systems, possibly cascaded, with adequate hand-crafted features. In constrast, we are interested in end-to-end architectures: these systems include several feature layers, with increasing abstraction at each layer. Compared to shallow systems, these feature layers are learnt for the task of interest, and do not require any engineering. We will show how neural networks are naturally well suited for end-to-end learning in NLP tasks. We will study multi-tasking different tasks, new semi-supervised learning techniques adapted to these deep architectures, and review end-to-end structured output learning. Finally, we will highlight how some of these advances can be applied to other fields of research, like computer vision, as well.
Author Information
Ronan Collobert (Apple)
Ronan Collobert received his master degree in pure mathematics from University of Rennes (France) in 2000. He then performed graduate studies in University of Montreal and IDIAP (Switzerland) under the Bengio brothers, and received his PhD in 2004 from University of Paris VI. He joined NEC Labs (USA) in January 2005 as a postdoc, and became a research staff member after about one year. His research interests always focused on large-scale machine-learning algorithms, with a particular interest in semi-supervised learning and deep learning architectures. Two years ago, his research shifted in the natural language processing area, slowly going towards automatic text understanding.
Jason E Weston (Meta AI)
Jason Weston received a PhD. (2000) from Royal Holloway, University of London under the supervision of Vladimir Vapnik. From 2000 to 2002, he was a researcher at Biowulf technologies, New York, applying machine learning to bioinformatics. From 2002 to 2003 he was a research scientist at the Max Planck Institute for Biological Cybernetics, Tuebingen, Germany. From 2004 to June 2009 he was a research staff member at NEC Labs America, Princeton. From July 2009 onwards he has been a research scientist at Google, New York. Jason Weston's current research focuses on various aspects of statistical machine learning and its applications, particularly in text and images.
More from the Same Authors
-
2020 : Invited Talk 4 Presentation - Jason Weston - (Towards) Learning from Conversing »
Jason E Weston -
2021 Spotlight: Hash Layers For Large Sparse Models »
Stephen Roller · Sainbayar Sukhbaatar · arthur szlam · Jason Weston -
2022 : Learning to Reason and Memorize with Self-Questioning »
Jack Lanchantin · Shubham Toshniwal · Jason E Weston · arthur szlam · Sainbayar Sukhbaatar -
2022 : Continuous Soft Pseudo-Labeling in ASR »
Tatiana Likhomanenko · Ronan Collobert · Navdeep Jaitly · Samy Bengio -
2023 Poster: Learning to Reason and Memorize with Self-Notes »
Jack Lanchantin · Shubham Toshniwal · Jason E Weston · arthur szlam · Sainbayar Sukhbaatar -
2022 : Invited Keynote by Jason Weston »
Jason Weston -
2022 : Learning to Reason and Memorize with Self-Questioning »
Jack Lanchantin · Shubham Toshniwal · Jason E Weston · arthur szlam · Sainbayar Sukhbaatar -
2022 Poster: Star Temporal Classification: Sequence Modeling with Partially Labeled Data »
Vineel Pratap · Awni Hannun · Gabriel Synnaeve · Ronan Collobert -
2022 Poster: Staircase Attention for Recurrent Processing of Sequences »
Da JU · Stephen Roller · Sainbayar Sukhbaatar · Jason E Weston -
2021 Poster: Hash Layers For Large Sparse Models »
Stephen Roller · Sainbayar Sukhbaatar · arthur szlam · Jason Weston -
2021 Poster: CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings »
Tatiana Likhomanenko · Qiantong Xu · Gabriel Synnaeve · Ronan Collobert · Alex Rogozhnikov -
2020 Workshop: Wordplay: When Language Meets Games »
Prithviraj Ammanabrolu · Matthew Hausknecht · Xingdi Yuan · Marc-Alexandre Côté · Adam Trischler · Kory Mathewson @korymath · John Urbanek · Jason Weston · Mark Riedl -
2020 : Panel »
Maxine Eskenazi · Ankur Parikh · Govindarajan Thattai · Alexander Rudnicky · Jason E Weston -
2020 : Invited Talk 4 Q/A - Jason Weston »
Jason E Weston -
2020 Memorial: In Memory of Olivier Chapelle »
Bernhard Schölkopf · Andre Elisseeff · Olivier Bousquet · Vladimir Vapnik · Jason E Weston -
2018 : Teaching through Dialogue and Games »
Jason E Weston -
2018 : Humans and models as embodied dialogue agents in text-based games »
Jason Weston -
2018 : The Conversational Intelligence Challenge 2 (ConvAI2) : Setup, Opening Words »
Jason Weston -
2016 : Jason Weston »
Jason E Weston -
2016 Workshop: Let's Discuss: Learning Methods for Dialogue »
Hal Daumé III · Paul Mineiro · Amanda Stent · Jason E Weston -
2016 Poster: Dialog-based Language Learning »
Jason E Weston -
2015 Workshop: Reasoning, Attention, Memory (RAM) Workshop »
Jason E Weston · Sumit Chopra · Antoine Bordes -
2015 : Evaluating Prerequisite Qualities For End-to-End Dialog Systems »
Jason E Weston -
2015 Poster: End-To-End Memory Networks »
Sainbayar Sukhbaatar · arthur szlam · Jason Weston · Rob Fergus -
2015 Poster: Learning to Segment Object Candidates »
Pedro O. Pinheiro · Ronan Collobert · Piotr Dollar -
2015 Spotlight: Learning to Segment Object Candidates »
Pedro O. Pinheiro · Ronan Collobert · Piotr Dollar -
2015 Oral: End-To-End Memory Networks »
Sainbayar Sukhbaatar · arthur szlam · Jason Weston · Rob Fergus -
2014 Workshop: 4th Workshop on Automated Knowledge Base Construction (AKBC) »
Sameer Singh · Fabian M Suchanek · Sebastian Riedel · Partha Pratim Talukdar · Kevin Murphy · Christopher Ré · William Cohen · Tom Mitchell · Andrew McCallum · Jason E Weston · Ramanathan Guha · Boyan Onyshkevych · Hoifung Poon · Oren Etzioni · Ari Kobren · Arvind Neelakantan · Peter Clark -
2011 Workshop: Learning Semantics »
Antoine Bordes · Jason E Weston · Ronan Collobert · Leon Bottou -
2011 Session: Oral Session 14 »
Ronan Collobert -
2011 Demonstration: SENNA Natural Language Processing Demo »
Ronan Collobert -
2010 Poster: Label Embedding Trees for Large Multi-Class Tasks »
Samy Bengio · Jason E Weston · David Grangier -
2009 Poster: Polynomial Semantic Indexing »
Bing Bai · Jason E Weston · David Grangier · Ronan Collobert · Kunihiko Sadamasa · Yanjun Qi · Corinna Cortes · Mehryar Mohri