Analysis of Variational Bayesian Latent Dirichlet Allocation: Weaker Sparsity than MAP
Abstract
Latent Dirichlet allocation (LDA) is a popular generative model of various objects such as texts and images, where an object is expressed as a mixture of latent topics. In this paper, we theoretically investigate variational Bayesian (VB) learning in LDA. More specifically, we analytically derive the leading term of the VB free energy under an asymptotic setup, and show that there exist transition thresholds in Dirichlet hyperparameters around which the sparsity-inducing behavior drastically changes. Then we further theoretically reveal the notable phenomenon that VB tends to induce weaker sparsity than MAP in the LDA model, which is opposed to other models. We experimentally demonstrate the practical validity of our asymptotic theory on real-world Last.FM music data.
Cite
Text
Nakajima et al. "Analysis of Variational Bayesian Latent Dirichlet Allocation: Weaker Sparsity than MAP." Neural Information Processing Systems, 2014.Markdown
[Nakajima et al. "Analysis of Variational Bayesian Latent Dirichlet Allocation: Weaker Sparsity than MAP." Neural Information Processing Systems, 2014.](https://mlanthology.org/neurips/2014/nakajima2014neurips-analysis/)BibTeX
@inproceedings{nakajima2014neurips-analysis,
title = {{Analysis of Variational Bayesian Latent Dirichlet Allocation: Weaker Sparsity than MAP}},
author = {Nakajima, Shinichi and Sato, Issei and Sugiyama, Masashi and Watanabe, Kazuho and Kobayashi, Hiroko},
booktitle = {Neural Information Processing Systems},
year = {2014},
pages = {1224-1232},
url = {https://mlanthology.org/neurips/2014/nakajima2014neurips-analysis/}
}