When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty
Abstract
One of challenges of artificial intelligence as a whole is robustness. Many issues such as adversarial examples, out of distribution performance, Clever Hans phenomena, and the wider areas of AI evaluation and explainable AI, have to do with the following question: Did the system fail because it is a hard instance or because something else? In this paper we address this question with a generic method for estimating IRT-based instance difficulty for a wide range of AI domains covering several areas, from supervised feature-based classification to automated reasoning. We show how to estimate difficulty systematically using off-the-shelf machine learning regression models. We illustrate the usefulness of this estimation for a range of applications.
Cite
Text
Martínez-Plumed et al. "When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty." AAAI Conference on Artificial Intelligence, 2022. doi:10.1609/AAAI.V36I7.20739Markdown
[Martínez-Plumed et al. "When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty." AAAI Conference on Artificial Intelligence, 2022.](https://mlanthology.org/aaai/2022/martinezplumed2022aaai-ai/) doi:10.1609/AAAI.V36I7.20739BibTeX
@inproceedings{martinezplumed2022aaai-ai,
title = {{When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty}},
author = {Martínez-Plumed, Fernando and Falcón, David Castellano and Aranda, Carlos Monserrat and Hernández-Orallo, José},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2022},
pages = {7719-7727},
doi = {10.1609/AAAI.V36I7.20739},
url = {https://mlanthology.org/aaai/2022/martinezplumed2022aaai-ai/}
}