Exact Model Averaging with Naive Bayesian Classifiers

Abstract

The naive classifier is a well-established mathematical model whose simplicity, speed and accuracy have made it a popular choice for classification in AI and engineering. In this paper we show that, given N features of interest, it is possible to perform tractable exact model averaging (MA) over all 2N pos-sible feature-set models. In fact, we show that it is possible to calculate parameters for a single naive classifier C ∗ such that C∗ produces predictions equivalent to those ob-tained by the full model-averaging, and we show that C ∗ can be constructed using the same time and space complexity required to construct a single naive classifier with MAP parameters. We present experimental results which show that on average the MA classi-fier typically outperforms the MAP classifier on simulated data, and we characterize how the relative performance varies with number of variables, number of training records, and complexity of the generating distribution. Fi-nally, we examine the performance of the MA naive model on the real-world ALARM and HEPAR networks and show MA improved classification here as well. 1.