Timezone: »

 
Poster
Diffusion-LM Improves Controllable Text Generation
Xiang Li · John Thickstun · Ishaan Gulrajani · Percy Liang · Tatsunori Hashimoto

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #606
in Poster Session 3 »

Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation. While recent works have demonstrated successes on controlling simple sentence attributes (e.g., sentiment), there has been little progress on complex, fine-grained controls (e.g., syntactic structure). To address this challenge, we develop a new non-autoregressive language model based on continuous diffusions that we call Diffusion-LM. Building upon the recent successes of diffusion models in continuous domains, Diffusion-LM iteratively denoises a sequence of Gaussian vectors into word vectors, yielding a sequence of intermediate latent variables. The continuous, hierarchical nature of these intermediate variables enables a simple gradient-based algorithm to perform complex, controllable generation tasks. We demonstrate successful control of Diffusion-LM for six challenging fine-grained control tasks, significantly outperforming prior work.

Keywords: language model · Diffusion model · controlled generation · controllable text generation · infilling

Author Information

Xiang Li (Stanford University)
John Thickstun (Stanford University)
Ishaan Gulrajani (MIT)
Percy Liang (Stanford University)
Tatsunori Hashimoto (Stanford)

More from the Same Authors