ML Anthology

The "Softmax" Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element

I. M. Elfadel, J. L. Wyatt Jr.
NeurIPS 1993 pp. 882-887
/neurips/1993/elfadel1993neurips-softmax/

Abstract

We use mean-field theory methods from Statistical Mechanics to derive the "softmax" nonlinearity from the discontinuous winner(cid:173) take-all (WTA) mapping. We give two simple ways of implementing "soft max" as a multiterminal network element. One of these has a number of important network-theoretic properties. It is a recipro(cid:173) cal, passive, incrementally passive, nonlinear, resistive multitermi(cid:173) nal element with a content function having the form of information(cid:173) theoretic entropy. These properties should enable one to use this element in nonlinear RC networks with such other reciprocal el(cid:173) ements as resistive fuses and constraint boxes to implement very high speed analog optimization algorithms using a minimum of hardware.

PDF NeurIPS Semantic Scholar

Cite

Text

Elfadel and Jr.. "The "Softmax" Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element." Neural Information Processing Systems, 1993.

Markdown

[Elfadel and Jr.. "The "Softmax" Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element." Neural Information Processing Systems, 1993.](https://mlanthology.org/neurips/1993/elfadel1993neurips-softmax/)

BibTeX

@inproceedings{elfadel1993neurips-softmax,
  title     = {{The "Softmax" Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element}},
  author    = {Elfadel, I. M. and Jr., J. L. Wyatt},
  booktitle = {Neural Information Processing Systems},
  year      = {1993},
  pages     = {882-887},
  url       = {https://mlanthology.org/neurips/1993/elfadel1993neurips-softmax/}
}