Generalized Agreement Statistics over Fixed Group of Experts
Abstract
Generalizations of chance corrected statistics to measure inter-expert agreement on class label assignments to the data instances have traditionally relied on the marginalization argument over a variable group of experts. Further, this argument has also resulted in agreement measures to evaluate the class predictions by an isolated classifier against the (multiple) labels assigned by the group of experts. We show that these measures are not necessarily suitable for application in the more typical fixed experts’ group scenario. We also propose novel, more meaningful, less variable generalizations for quantifying both the inter-expert agreement over the fixed group and assessing a classifier’s output against it in a multi-expert multi-class scenario by taking into account expert-specific biases and correlations.
Cite
Text
Shah. "Generalized Agreement Statistics over Fixed Group of Experts." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2011. doi:10.1007/978-3-642-23808-6_13Markdown
[Shah. "Generalized Agreement Statistics over Fixed Group of Experts." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2011.](https://mlanthology.org/ecmlpkdd/2011/shah2011ecmlpkdd-generalized/) doi:10.1007/978-3-642-23808-6_13BibTeX
@inproceedings{shah2011ecmlpkdd-generalized,
title = {{Generalized Agreement Statistics over Fixed Group of Experts}},
author = {Shah, Mohak},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2011},
pages = {191-206},
doi = {10.1007/978-3-642-23808-6_13},
url = {https://mlanthology.org/ecmlpkdd/2011/shah2011ecmlpkdd-generalized/}
}