Attribute Dependencies, Understandability and Split Selection in Tree Based Models

Abstract

The attributes' interdependencies have strong effect on understandability of tree based models. If strong dependencies between the attributes are not recognized and these attributes are not used as splits near the root of the tree this causes node replications in lower levels of the tree, blurs the description of dependencies and also might cause drop of accuracy. If Relief family of algorithms which is capable of estimating the attributes' dependencies is used for split selectors we can partly overcome the problem. However, typically we still want to optimize accuracy of the tree and therefore use accuracy as the split selector measure near the fringe of the tree. We present a technique which helps us select a split criterion during tree growing based on some theoretical properties of Relief's estimate. We support our claims with empirical results.