Large Scale Knowledge Base Systems: An Empirical Evaluation Perspective

Abstract

In this paper, we discuss how our work on evaluating Semantic Web knowledge base systems (KBSs) contributes to address some broader AI problems. First, we show how our approach provides a benchmarking solution to the Semantic Web, a new application area of AI. Second, we discuss how the approach is also beneficial in a more traditional AI context. We focus on issues such as scalability, performance tradeoffs, and the comparison of different classes of systems. Benchmarking Semantic Web KBSs Our research interest is to develop objective and unbiased ways to evaluate Semantic Web knowledge base systems (KBSs) (See Guo, Pan and Heflin 2004). Specifically, we have conducted research on benchmarking KBSs that store, reason and query statements described in OWL1, which is a standard language for describing and publishing Web ontologies. As a product of our work, we have developed the Lehigh University Benchmark (LUBM). The LUBM is, to the best of our knowledge, the first of its kind and has become well recognized in the Semantic Web community. The LUBM is designed to fill a void that we consider particularly important, i.e., the evaluation of systems with respect to large instance data that commit to an ontology of moderate size. In creating the benchmark, we have developed: 1) An OWL ontology for the university domain. 2) A technique for synthetically generating instance data over that ontology. Importantly, this data can be regenerated given only a seed and can be scaled to an arbitrary size. Moreover, to make it as realistic as possible, the data is generated by obeying to a set of restrictions that are elicited from an investigation into the domain (e.g. the ratios between instances of different classes and the cardinality of different properties for individuals of different types). 3) Fourteen test queries against the instance data. These queries have been chosen to represent a variety of