Novel Applications of Item Response Theory for Analysing Data Set Complexity and Benchmark Selection

Abstract

Item response theory (IRT) was developed in psychometrics to measure the latent skills of human respondents based on their observed responses to items with different difficulty levels. Human ability is high in IRT when one correctly responds to difficult items despite random mistakes in easy items. IRT has been recently framed as a powerful tool to characterise instance hardness in classification problems by measuring difficulty and discrimination levels of instances in a data set based on the correctness of a set of classifiers. Here, we generalise such a concept to the data set level by taking a pool of 509 classification data sets and assessing their difficulties and discriminations based on the performance achieved by 95 classifiers when solving these problems. The ability is estimated such that high abilities are assigned to classifiers with better behaviour in hard data sets. We further evaluated IRT in two distinct applications. First, we build a regression meta-model where complexity measures are used to predict the IRT parameters of new data sets without the need to retrain the IRT model. Second, we propose two IRT-based benchmarks with 30 data sets each to test classifiers, one selected for diversity and another selected for greater difficulty. Both benchmarks may be used to evaluate new methods more broadly, instead of the common practice of gathering random data sets from public repositories.