On the Use of Evidence in Neural Networks
Abstract
The Bayesian "evidence" approximation has recently been employed to determine the noise and weight-penalty terms used in back-propagation. This paper shows that for neural nets it is far easier to use the exact result than it is to use the evidence approximation. Moreover, unlike the evi(cid:173) dence approximation, the exact result neither has to be re-calculated for every new data set, nor requires the running of computer code (the exact result is closed form). In addition, it turns out that the evidence proce(cid:173) dure's MAP estimate for neural nets is, in toto, approximation error. An(cid:173) other advantage of the exact analysis is that it does not lead one to incor(cid:173) rect intuition, like the claim that using evidence one can "evaluate differ(cid:173) ent priors in light of the data". This paper also discusses sufficiency conditions for the evidence approximation to hold, why it can sometimes give "reasonable" results, etc.
Cite
Text
Wolpert. "On the Use of Evidence in Neural Networks." Neural Information Processing Systems, 1992.Markdown
[Wolpert. "On the Use of Evidence in Neural Networks." Neural Information Processing Systems, 1992.](https://mlanthology.org/neurips/1992/wolpert1992neurips-use/)BibTeX
@inproceedings{wolpert1992neurips-use,
title = {{On the Use of Evidence in Neural Networks}},
author = {Wolpert, David},
booktitle = {Neural Information Processing Systems},
year = {1992},
pages = {539-546},
url = {https://mlanthology.org/neurips/1992/wolpert1992neurips-use/}
}