Regularization Learning of Neural Networks for Generalization

Abstract

In this paper, we propose a learning method of neural networks based on the regularization method and analyze its generalization capability. In learning from examples, training samples are independently drawn from some unknown probability distribution. The goal of learning is minimizing the expected risk for future test samples, which are also drawn from the same distribution. The problem can be reduced to estimating the probability distribution with only samples, but it is generally ill-posed. In order to solve it stably, we use the regularization method. Regularization learning can be done in practice by increasing samples by adding appropriate amount of noise to the training samples. We estimate its generalization error, which is defined as a difference between the expected risk accomplished by the learning and the truly minimum expected risk. Assume p -dimensional density function is s -times differentiable for any variable. We show the mean square of the generalization error of regularization learning is given as Dn ^−2 s /(2s s+p ) where n is the number of samples and D is a constant dependent on the complexity of the neural network and the difficulty of the problem.