TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings
Abstract
This paper presents the Text Encoding Diffusion Model (TEncDM), a novel approach to diffusion modeling that operates in the space of pre-trained language model encodings. In contrast to traditionally used embeddings, encodings integrate contextual information. In our approach, we also employ a transformer-based decoder, specifically designed to incorporate context in the token prediction process. We conduct a comprehensive examination of the influence of the encoder, decoder, noise scheduler, and self-conditioning on zero-shot generation. Furthermore, we compare TEncDM with previous approaches on three conditional text generation tasks: QQP, XSum, and Wiki-Auto. The results show that TEncDM exhibits superior performance compared to existing non-autoregressive diffusion models.
Cite
Text
Shabalin et al. "TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings." AAAI Conference on Artificial Intelligence, 2025. doi:10.1609/AAAI.V39I23.34696Markdown
[Shabalin et al. "TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings." AAAI Conference on Artificial Intelligence, 2025.](https://mlanthology.org/aaai/2025/shabalin2025aaai-tencdm/) doi:10.1609/AAAI.V39I23.34696BibTeX
@inproceedings{shabalin2025aaai-tencdm,
title = {{TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings}},
author = {Shabalin, Alexander and Meshchaninov, Viacheslav and Chimbulatov, Egor and Lapikov, Vladislav and Kim, Roman and Bartosh, Grigory and Molchanov, Dmitry and Markov, Sergey and Vetrov, Dmitry P.},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2025},
pages = {25110-25118},
doi = {10.1609/AAAI.V39I23.34696},
url = {https://mlanthology.org/aaai/2025/shabalin2025aaai-tencdm/}
}