Timezone: »

 
Poster
Discrete Flows: Invertible Generative Models of Discrete Data
Dustin Tran · Keyon Vafa · Kumar Agrawal · Laurent Dinh · Ben Poole

Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #112
in Deep Learning -- Generative Models »

While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We consider two flow architectures: discrete autoregressive flows that enable bidirectionality, allowing, for example, tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows that enable efficient non-autoregressive generation as in RealNVP. Empirically, we find that discrete autoregressive flows outperform autoregressive baselines on synthetic discrete distributions, an addition task, and Potts models; and bipartite flows can obtain competitive performance with autoregressive baselines on character-level language modeling for Penn Tree Bank and text8.

Keywords: Generative Models · Deep Learning
[ Paper]  [ Poster] 

Author Information

Dustin Tran (Google Brain)
Keyon Vafa (Columbia University)
Kumar Agrawal (Google AI Resident)
Laurent Dinh (Google Brain)
Ben Poole (Google Brain)

More from the Same Authors