Weight Space Probability Densities in Stochastic Learning: II. Transients and Basin Hopping Times
Abstract
In stochastic learning, weights are random variables whose time evolution is governed by a Markov process. At each time-step, n, the weights can be described by a probability density function pew, n). We summarize the theory of the time evolution of P, and give graphical examples of the time evolution that contrast the behavior of stochastic learning with true gradient descent (batch learning). Finally, we use the formalism to obtain predictions of the time required for noise-induced hopping between basins of different optima. We compare the theoretical predictions with simulations of large ensembles of networks for simple problems in supervised and unsupervised learning.
Cite
Text
Orr and Leen. "Weight Space Probability Densities in Stochastic Learning: II. Transients and Basin Hopping Times." Neural Information Processing Systems, 1992.Markdown
[Orr and Leen. "Weight Space Probability Densities in Stochastic Learning: II. Transients and Basin Hopping Times." Neural Information Processing Systems, 1992.](https://mlanthology.org/neurips/1992/orr1992neurips-weight/)BibTeX
@inproceedings{orr1992neurips-weight,
title = {{Weight Space Probability Densities in Stochastic Learning: II. Transients and Basin Hopping Times}},
author = {Orr, Genevieve B. and Leen, Todd K.},
booktitle = {Neural Information Processing Systems},
year = {1992},
pages = {507-514},
url = {https://mlanthology.org/neurips/1992/orr1992neurips-weight/}
}