Mutual Information in Learning Feature Transformations

Abstract

We present feature transformations useful for exploratory data analysis or for pattern recognition. Transformations are learned from example data sets by maximizing the mutual information between transformed data and their class labels. We make use of Renyi's quadratic entropy, and we extend the work of Principe et al. to mutual information between continuous multidimensional variables and discrete-valued class labels. 1. Introduction Reducing the dimensionality of feature vectors is usually an essential step in pattern recognition tasks to achieve practical feasibility. Usually this is done by using domain knowledge or heuristics. Dimensionality reduction is also essential in exploratory data analysis, where the purpose often is to map data onto a low-dimensional space for human eyes to gain some insight to the data. It is well known that principal component analysis (PCA) has nothing to do with discriminative features optimal for classification, since it is only concerned ...