Algebraic Analysis for Singular Statistical Estimation
Abstract
This paper clarifies learning efficiency of a non-regular parametric model such as a neural network whose true parameter set is an analytic variety with singular points. By using Sato’s b-function we rigorously prove that the free energy or the Bayesian stochastic complexity is asymptotically equal to λ _1 log n − ( m _1 − 1) log log n +constant, where λ _1 is a rational number, m _1 is a natural number, and n is the number of training samples. Also we show an algorithm to calculate λ _1 and m _1 based on the resolution of singularity. In regular models, 2 λ _1 is equal to the number of parameters and m _1 = 1, whereas in non-regular models such as neural networks, 2 λ _1 is smaller than the number of parameters and m _1 ≥ 1.
Cite
Text
Watanabe. "Algebraic Analysis for Singular Statistical Estimation." International Conference on Algorithmic Learning Theory, 1999. doi:10.1007/3-540-46769-6_4Markdown
[Watanabe. "Algebraic Analysis for Singular Statistical Estimation." International Conference on Algorithmic Learning Theory, 1999.](https://mlanthology.org/alt/1999/watanabe1999alt-algebraic/) doi:10.1007/3-540-46769-6_4BibTeX
@inproceedings{watanabe1999alt-algebraic,
title = {{Algebraic Analysis for Singular Statistical Estimation}},
author = {Watanabe, Sumio},
booktitle = {International Conference on Algorithmic Learning Theory},
year = {1999},
pages = {39-50},
doi = {10.1007/3-540-46769-6_4},
url = {https://mlanthology.org/alt/1999/watanabe1999alt-algebraic/}
}