Deconstructing the Digit Recognition Problem

Abstract

Decision tree pruning techniques and other forms of overfitting avoidance have often been considered statistical means of improving predictive accuracy. Intuitively, they are intended to determine the appropriate level of complexity for an induced model by distinguishing between signal and noise in training data, patterns that reflect the true underlying nature of data generation and those that arise by chance. In fact, however, overfitting avoidance methods simply encode preferences for certain classes of models. These preferences may increase accuracy when they bias induction in favor of predictive models, but they may just as well have the opposite effect if they bias induction away from predictive models. This paper analyzes the conditional value of overfitting avoidance and focuses on the effect of one highly-regarded pruning technique in variations of the well-known digit recognition problem.