Timezone: »

 
Poster
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Tong Wu · Zhihao Fan · Xiao Liu · Hai-Tao Zheng · Yeyun Gong · yelong shen · Jian Jiao · Juntao Li · zhongyu wei · Jian Guo · Nan Duan · Weizhu Chen

Tue Dec 12 03:15 PM -- 05:15 PM (PST) @ Great Hall & Hall B1+B2 #325
in Poster Session 2 »
Event URL: https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion »
Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach.To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right.In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models and that it can be $100\times\sim600\times$ faster when achieving comparable results. Our code is available at https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion.

Author Information

Tong Wu (Tsinghua University)
Zhihao Fan (Alibaba Damo Academy)
Xiao Liu (Microsoft Research Asia)
Xiao Liu

My name is Xiao LIU (刘啸 in Chinese). I am a researcher in the [Natural Language Computing](https://www.microsoft.com/en-us/research/group/natural-language-computing/) group at [MSRA](https://www.msra.cn/). I obtained my Ph.D. degree from [BIT](http://www.bit.edu.cn/) in June 2022, under the supervision of Prof. [Heyan Huang](http://cs.bit.edu.cn/szdw/jsml/js/hhy/index.htm). My doctoral thesis is about event extraction and was awarded the excellent Ph.D. thesis of Beijing Institute of Technology. My current research interests include **generation**, **retrieval** and **extraction**. You can find me in [Microsoft Research](https://www.microsoft.com/en-us/research/people/xiaoliu2/).

Hai-Tao Zheng (Tsinghua University, Tsinghua University)
Yeyun Gong (Microsoft)
yelong shen (Microsoft)
Jian Jiao (Microsoft)
Juntao Li (Soochow University, China)
zhongyu wei
Jian Guo
Nan Duan (Microsoft Research Asia)
Weizhu Chen (Microsoft Azure AI)

More from the Same Authors