Partial Occam's Razor and Its Applications

Abstract

We introduce the notion of “partial Occam algorithm”. A partial Occam algorithm produces a succinct hypothesis that is partially consistent with given examples, where the proportion of consistent examples is a bit more than half. By using this new notion, we propose one approach for obtaining a PAC learning algorithm: A partial Occam algorithm is equivalent to a weak PAC learning algorithm. Thus, by using boosting techniques, we can obtain an ordinary PAC learning algorithm from this weak PAC learning algorithm. We demonstrate with some examples that some improvement is possible by this approach, in particular in the hypothesis size. First, we obtain a non proper PAC learning algorithm for k -DNF, which has similar sample complexity as Littlestone's Winnow, but produces hypothesis of size polynomial in d and log k for a k -DNF target with n variables and d terms. ( Cf . The hypothesis size of Winnow is O ( n ^k).) Next we show that 1-decision lists of length d with n variables are non-proper PAC learn able by using O (1 (log + 16^d log n ( d + log log n )^2)) examples within polynomial time w.r.t. n , 2^d, 1/ε, and log 1/S. Again, while this sample complexity is similar to Winnow, we improve the hypothesis size. We also point out that our algorithms are robust against random classification noise.