KITSS: A Knowledge-Based Translation System for Test Scenarios

Abstract

Machine-assisted language translation systems for technical documents, guide humans through a process of selecting and composing variant partial translations. The constrained nature of technical sublanguages makes language processing aids cost-effective to build and use. Analogously, we have developed KITSS, a knowledge-based translation system for converting informal English scenarios of the desired behavior of complex reactive systems into formal, executable test scripts. A trainable parser and reference resolver capture domain-specific linguistic knowledge. A logic analyzer establishes coherence in the translation process in a role comparable to a story understander. It checks the consistency of each step of a translated test script using a theorem prover, a planner, and logic-encoded background knowledge about the system under test. This helps correct common but serious specification errors, including underspecificity, omitted steps, and even some outright mis-statements. To evaluate how well such technology can scale, we have exercised our technology progressively on a graduated corpus of 100 behavior scenarios spanning 7 advanced calling features for a private telephone switch (PBX), successfully translating 70% into test scripts without any manual post-hoc editing. Our experience with KITSS has enabled us to identify many of the tradeoffs in accommodating informality in specification, versus demanding formality from a human agent.