Dynamically-Adaptive Winner-Take-All Networks

Abstract

Winner-Take-All (WTA) networks. in which inhibitory interconnec(cid:173) tions are used to determine the most highly-activated of a pool of unilS. are an important part of many neural network models. Unfortunately, convergence of normal WT A networks is extremely sensitive to the magnitudes of their weights, which must be hand-tuned and which gen(cid:173) erally only provide the right amount of inhibition across a relatively small range of initial conditions. This paper presents Dynamjcally(cid:173) Adaptive Winner-Telke-All (DA WTA) netw<rls, which use a regulatory unit to provide the competitive inhibition to the units in the network. The DA WT A regulatory unit dynamically adjusts its level of activation during competition to provide the right amount of inhibition to differ(cid:173) entiate between competitors and drive a single winner. This dynamic adaptation allows DA WT A networks to perform the winner-lake-all function for nearly any network size or initial condition. using O(N) connections. In addition, the DA WT A regulaaory unit can be biased 10 find the level of inhibition necessary to settle upon the K most highly(cid:173) activated units, and therefore serve as a K -Winners-Take-All network.