Balancing Between Bagging and Bumping

Abstract

We compare different methods to combine predictions from neu(cid:173) ral networks trained on different bootstrap samples of a regression problem. One of these methods, introduced in [6] and which we here call balancing, is based on the analysis of the ensemble gen(cid:173) eralization error into an ambiguity term and a term incorporating generalization performances of individual networks. We show how to estimate these individual errors from the residuals on valida(cid:173) tion patterns. Weighting factors for the different networks follow from a quadratic programming problem. On a real-world problem concerning the prediction of sales figures and on the well-known Boston housing data set, balancing clearly outperforms other re(cid:173) cently proposed alternatives as bagging [1] and bumping [8].