Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria
Abstract
The ensemble dynamics of stochastic learning algorithms can be studied using theoretical techniques from statistical physics. We develop the equations of motion for the weight space probability densities for stochastic learning algorithms. We discuss equilibria in the diffusion approximation and provide expressions for special cases of the LMS algorithm. The equilibrium densities are not in general thermal (Gibbs) distributions in the objective function be(cid:173) ing minimized, but rather depend upon an effective potential that includes diffusion effects. Finally we present an exact analytical expression for the time evolution of the density for a learning algo(cid:173) rithm with weight updates proportional to the sign of the gradient.
Cite
Text
Leen and Moody. "Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria." Neural Information Processing Systems, 1992.Markdown
[Leen and Moody. "Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria." Neural Information Processing Systems, 1992.](https://mlanthology.org/neurips/1992/leen1992neurips-weight/)BibTeX
@inproceedings{leen1992neurips-weight,
title = {{Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria}},
author = {Leen, Todd K. and Moody, John E.},
booktitle = {Neural Information Processing Systems},
year = {1992},
pages = {451-458},
url = {https://mlanthology.org/neurips/1992/leen1992neurips-weight/}
}