Nested Joint Probability Model for Morphological Analysis and Its Grid Pruning

Abstract

In recent work on morphological analysis based on statistical models the conditional probability of the observed i-th word Wi with the i-th tag ti after the (i-1)-th tag ti-1 is defined as the product of observation symbol probability and the state transition probability (i.e. P(Wi|ti)?P(ti|ti-1)). In order to improve accuracy, we face the following problems: 1) If we build hidden state levels using stricter categories (e.g. lowest POS class, over 3-gram, or word themselves), the state transition probability matrix becomes much bigger and more sparse; 2) If we use rough categories the reliability of statistical information becomes lower in some parts of speech; and 3) the best state level is not the same among POS category, and some heuristic knowledge is necessary to select the best state structure.