Learning Sequential Structure in Simple Recurrent Networks
Abstract
We explore a network architecture introduced by Elman (1988) for predicting successive elements of a sequence. The network uses the pattern of activation over a set of hidden units from time-step t-l, together with element t, to predict element t+ 1. When the network is trained with strings from a particular finite-state grammar, it can learn to be a perfect finite-state recognizer for the grammar. Cluster analyses of the hidden-layer patterns of activation showed that they encode prediction-relevant information about the entire path traversed through the network. We illustrate the phases of learning with cluster analyses performed at different points during training.
Cite
Text
Servan-Schreiber et al. "Learning Sequential Structure in Simple Recurrent Networks." Neural Information Processing Systems, 1988.Markdown
[Servan-Schreiber et al. "Learning Sequential Structure in Simple Recurrent Networks." Neural Information Processing Systems, 1988.](https://mlanthology.org/neurips/1988/servanschreiber1988neurips-learning/)BibTeX
@inproceedings{servanschreiber1988neurips-learning,
title = {{Learning Sequential Structure in Simple Recurrent Networks}},
author = {Servan-Schreiber, David and Cleeremans, Axel and McClelland, James L.},
booktitle = {Neural Information Processing Systems},
year = {1988},
pages = {643-652},
url = {https://mlanthology.org/neurips/1988/servanschreiber1988neurips-learning/}
}