Timezone: »

 
Poster
Deliberation Networks: Sequence Generation Beyond One-Pass Decoding
Yingce Xia · Fei Tian · Lijun Wu · Jianxin Lin · Tao Qin · Nenghai Yu · Tie-Yan Liu

Mon Dec 04 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #127
in Posters Mon »

The encoder-decoder framework has achieved promising progress for many sequence generation tasks, including machine translation, text summarization, dialog system, image captioning, etc. Such a framework adopts an one-pass forward process while decoding and generating a sequence, but lacks the deliberation process: A generated sequence is directly used as final output without further polishing. However, deliberation is a common behavior in human's daily life like reading news and writing papers/articles/books. In this work, we introduce the deliberation process into the encoder-decoder framework and propose deliberation networks for sequence generation. A deliberation network has two levels of decoders, where the first-pass decoder generates a raw sequence and the second-pass decoder polishes and refines the raw sentence with deliberation. Since the second-pass deliberation decoder has global information about what the sequence to be generated might be, it has the potential to generate a better sequence by looking into future words in the raw sentence. Experiments on neural machine translation and text summarization demonstrate the effectiveness of the proposed deliberation networks. On the WMT 2014 English-to-French translation task, our model establishes a new state-of-the-art BLEU score of 41.5.

Keywords: Recurrent Networks · Deep Learning
[ Paper] 

Author Information

Yingce Xia (Microsoft Research)
Fei Tian (Miicrosoft Research)
Lijun Wu (Sun Yat-sen University)
Jianxin Lin (USTC)
Tao Qin (Microsoft Research)
Nenghai Yu (University of Science and Technology of China)
Tie-Yan Liu (Microsoft Research)

More from the Same Authors