Horizonal Generalization

Abstract

In conventional supervised learning, one searches for “vertical” patterns, coupling inputs directly to outputs. One can instead search for “horizontal” patterns, which go across the input space, coupling output values on one part of the input space with output values on another. One way to do this is to pre-process the problem, in a manner similar to the “embedding” process of non-linear time-scries analysis. The training set produced by this pre-processing is constructed solely from the output components of the original training set. (The input components of the original training set are used in concert with cross-validation to determine the details of the processing of those output components.) This paper presents three sets of experiments of the efficacy of such pre-processing, involving numerical, Boolean, and visual tasks. The first set involves small (8 bit) problems. In all but one of the 36 experiments in that set, the pre-processing improved the generalization performance, often inducing perfect generalization. The average ratio of the generalization error rate with the pre-proccssing to the error rate without it is .23, +/- .05. The second set of experiments involve a 24-bit input space. For a number of different target functions, a training set is actively constructed by sampling the target function at 301 pre-specified points in that space. A decision-directed version of the pre-processing is then used to extrapolate from that training set to the remaining 224 – 301 points in the input space. The average error rate across here is 8.6%. The third set of experiments is a variation of the robot arm problem recently investigated by MacKay (1992). Here the rms error rate (for extrapolation) was only 3 times the noise level.