Variational Methods for the Dirichlet Process

Abstract

Variational inference methods, including mean field methods and loopy beliefpropagation, have been widely used for approximate probabilistic inference ingraphical models. While often less accurate than MCMC, variational methodsprovide a fast deterministic approximation to marginal and conditionalprobabilities. Such approximations can be particularly useful in highdimensional problems where sampling methods are too slow to be effective. Alimitation of current methods, however, is that they are restricted toparametric probabilistic models. MCMC does not have such a limitation;indeed, MCMC samplers have been developed for the Dirichlet process (DP), anonparametric distribution on distributions (Ferguson, 1973) that is thecornerstone of Bayesian nonparametric statistics~\citep (Escobar and West,1995; Neal, 2000}. In this paper, we develop a mean-field variationalapproach to approximate inference for the Dirichlet process, where theapproximate posterior is based on the truncated stick-breaking construction(Ishwaran and James, 2001). We compare our approach to DP samplers for Gaussian DP mixture models.