Combination of Inductive Mondrian Conformal Predictors

Abstract

It is well known that ensembling predictions from different Machine Learning (ML) algorithms can improve accuracy. This paper proposes a approach to combine Conformal Predictors (CPs) with different underlying ML algorithms in a way that preserves their key property, i.e. validity. Different combination methods are discussed and their performance is evaluated on a chemoinformatics problem. In order to deal with the size, high-dimensionality, and strong imbalance of the data set, the paper applies a special type of CP: an Inductive Mondrian Conformal Predictor. We propose and evaluate, alongside methods from Statistical Hypothesis Testing, a heuristically motivated method for learning to combine CPs to improve the quality of prediction. We also explore a general nonparametric method for recovering validity after combination using a calibration set. On a real-world data set, several of the combined predictors consistently outperform the base CPs.