Timezone: »

 
Poster
Deep State Space Models for Unconditional Word Generation
Florian Schmidt · Thomas Hofmann

Thu Dec 06 07:45 AM -- 09:45 AM (PST) @ Room 210 #61
in Thu Poster Session A »

Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuances. We show that a non-autoregressive deep state space model with a clear separation of global and local uncertainty can be built from only two ingredients: An independent noise source and a deterministic transition function. Recent advances on flow-based variational inference can be used to train an evidence lower-bound without resorting to annealing, auxiliary losses or similar measures. The result is a highly interpretable generative model on par with comparable auto-regressive models on the task of word generation.

Keywords: Natural Language Processing · Latent Variable Models

Author Information

Florian Schmidt (ETH Zürich)
Thomas Hofmann (ETH Zurich)

More from the Same Authors