Parsing Embedded Clauses with Distributed Neural Networks

Abstract

A distributed neural network model called SPEC for processing sentences with recursive relative clauses is described. The model is based on sep-arating the tasks of segmenting the input word sequence into clauses, forming the case-role rep-resentations, and keeping track of the recursive embeddings into different modules. The system needs to be trained only with the basic sentence constructs, and it generalizes not only to new in-stances of familiar relative clause structures, but to novel structures as well. SPEC exhibits plausi-ble memory degradation as the depth of the center embeddings increases, its memory is primed by earlier constituents, and its performance is aided by semantic constraints between the constituents. The ability to process structure is largely due to a central executive network that monitors and con-trols the execution of the entire system. This way, in contrast to earlier subsymbolic systems, parsing is modeled as a controlled high-level pro-cess rather than one based on automatic reflex responses. The girl, who liked the dog, saw the boy’, and it will generalize to different versions of the same struc-ture, such as The dog, who bit the girl, chased the cat (Miikkulainen 1990). However, such a net-work cannot parse sentences with novel combina-tions of relative clauses, such as The girl, who liked the dog, saw the boy, who chased the cat. The problem is that distributed neural networks are pat-tern transformers, and they generalize by interpolating between patterns on which they were trained. They cannot make inferences by dynamically combining pro-cessing knowledge that was previously associated to dif-ferent contexts, such as processing a relative clause at a new place in an otherwise familiar sentence structure. This lack of generalization is a serious problem, given how effortlessly people can understand sentences they have never seen before.