Iterative Scaling and Coordinate Descent Methods for Maximum Entropy Models
Abstract
Maximum entropy (Maxent) is useful in natural language processing and many other areas. Iterative scaling (IS) methods are one of the most popular approaches to solve Maxent. With many variants of IS methods, it is difficult to understand them and see the differences. In this paper, we create a general and unified framework for iterative scaling methods. This framework also connects iterative scaling and coordinate descent methods. We prove general convergence results for IS methods and analyze their computational complexity. Based on the proposed framework, we extend a coordinate descent method for linear SVM to Maxent. Results show that it is faster than existing iterative scaling methods.
Cite
Text
Huang et al. "Iterative Scaling and Coordinate Descent Methods for Maximum Entropy Models." Journal of Machine Learning Research, 2010.Markdown
[Huang et al. "Iterative Scaling and Coordinate Descent Methods for Maximum Entropy Models." Journal of Machine Learning Research, 2010.](https://mlanthology.org/jmlr/2010/huang2010jmlr-iterative/)BibTeX
@article{huang2010jmlr-iterative,
title = {{Iterative Scaling and Coordinate Descent Methods for Maximum Entropy Models}},
author = {Huang, Fang-Lan and Hsieh, Cho-Jui and Chang, Kai-Wei and Lin, Chih-Jen},
journal = {Journal of Machine Learning Research},
year = {2010},
pages = {815-848},
volume = {11},
url = {https://mlanthology.org/jmlr/2010/huang2010jmlr-iterative/}
}