Discrete Flows: Invertible Generative Models of Discrete Data
Abstract
While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We display proofs of concept under 2 flow architectures: discrete autoregressive flows enable bidirectionality, allowing for example tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows (i.e., with layer structure from RealNVP) enable parallel generation such as exact nonautoregressive text modeling.
Cite
Text
Tran et al. "Discrete Flows: Invertible Generative Models of Discrete Data." ICLR 2019 Workshops: DeepGenStruct, 2019.Markdown
[Tran et al. "Discrete Flows: Invertible Generative Models of Discrete Data." ICLR 2019 Workshops: DeepGenStruct, 2019.](https://mlanthology.org/iclrw/2019/tran2019iclrw-discrete/)BibTeX
@inproceedings{tran2019iclrw-discrete,
title = {{Discrete Flows: Invertible Generative Models of Discrete Data}},
author = {Tran, Dustin and Vafa, Keyon and Agrawal, Kumar and Dinh, Laurent and Poole, Ben},
booktitle = {ICLR 2019 Workshops: DeepGenStruct},
year = {2019},
url = {https://mlanthology.org/iclrw/2019/tran2019iclrw-discrete/}
}