Discrete Flows: Invertible Generative Models of Discrete Data

Abstract

While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We display proofs of concept under 2 flow architectures: discrete autoregressive flows enable bidirectionality, allowing for example tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows (i.e., with layer structure from RealNVP) enable parallel generation such as exact nonautoregressive text modeling.

Cite

Text

Tran et al. "Discrete Flows: Invertible Generative Models of Discrete Data." ICLR 2019 Workshops: DeepGenStruct, 2019.

Markdown

[Tran et al. "Discrete Flows: Invertible Generative Models of Discrete Data." ICLR 2019 Workshops: DeepGenStruct, 2019.](https://mlanthology.org/iclrw/2019/tran2019iclrw-discrete/)

BibTeX

@inproceedings{tran2019iclrw-discrete,
  title     = {{Discrete Flows: Invertible Generative Models of Discrete Data}},
  author    = {Tran, Dustin and Vafa, Keyon and Agrawal, Kumar and Dinh, Laurent and Poole, Ben},
  booktitle = {ICLR 2019 Workshops: DeepGenStruct},
  year      = {2019},
  url       = {https://mlanthology.org/iclrw/2019/tran2019iclrw-discrete/}
}