InteractEva: A Simulation-Based Evaluation Framework for Interactive AI Systems
Abstract
Evaluating interactive AI (IAI) systems is a challenging task, as their output highly depends on the performed user actions. As a result, developers often depend on limited and mostly qualitative data derived from user testing to improve their systems. In this paper, we present InteractEva; a systematic evaluation framework for IAI systems. InteractEva employs (a) a user simulation backend to test the system against different use cases and user interactions at scale with (b) an interactive frontend allowing developers to perform important quantitative evaluation tasks, including acquiring a performance overview, performing error analysis, and conducting what-if studies. The framework has supported the evaluation and improvement of an industrial IAI text extraction system, results of which will be presented during our demonstration.
Cite
Text
Katsis et al. "InteractEva: A Simulation-Based Evaluation Framework for Interactive AI Systems." AAAI Conference on Artificial Intelligence, 2022. doi:10.1609/AAAI.V36I11.21721Markdown
[Katsis et al. "InteractEva: A Simulation-Based Evaluation Framework for Interactive AI Systems." AAAI Conference on Artificial Intelligence, 2022.](https://mlanthology.org/aaai/2022/katsis2022aaai-interacteva/) doi:10.1609/AAAI.V36I11.21721BibTeX
@inproceedings{katsis2022aaai-interacteva,
title = {{InteractEva: A Simulation-Based Evaluation Framework for Interactive AI Systems}},
author = {Katsis, Yannis and Hanafi, Maeda F. and Cooper, Martín Santillán and Li, Yunyao},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2022},
pages = {13182-13184},
doi = {10.1609/AAAI.V36I11.21721},
url = {https://mlanthology.org/aaai/2022/katsis2022aaai-interacteva/}
}