Timezone: »

Diffusion-LM Improves Controllable Text Generation
Xiang Li · John Thickstun · Ishaan Gulrajani · Percy Liang · Tatsunori Hashimoto

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #606

Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation. While recent works have demonstrated successes on controlling simple sentence attributes (e.g., sentiment), there has been little progress on complex, fine-grained controls (e.g., syntactic structure). To address this challenge, we develop a new non-autoregressive language model based on continuous diffusions that we call Diffusion-LM. Building upon the recent successes of diffusion models in continuous domains, Diffusion-LM iteratively denoises a sequence of Gaussian vectors into word vectors, yielding a sequence of intermediate latent variables. The continuous, hierarchical nature of these intermediate variables enables a simple gradient-based algorithm to perform complex, controllable generation tasks. We demonstrate successful control of Diffusion-LM for six challenging fine-grained control tasks, significantly outperforming prior work.

Author Information

Xiang Li (Stanford University)
John Thickstun (Stanford University)
Ishaan Gulrajani (MIT)
Percy Liang (Stanford University)
Tatsunori Hashimoto (Stanford)

More from the Same Authors