Aggregating Binary Judgments Ranked by Accuracy
Abstract
We revisit the fundamental problem of predicting a binary ground truth based on independent binary judgments provided by experts. When the accuracy levels of the experts are known, the problem can be solved easily through maximum likelihood estimation. We consider, however, a setting in which we are given only a ranking of the experts by their accuracy. Motivated by the worst-case approach to handle the missing information, we consider three objective functions and design efficient algorithms for optimizing them. In particular, the recently popular distortion objective leads to an intuitive new rule. We show that our algorithms perform well empirically using real and synthetic data in collaborative filtering and political prediction domains.
Cite
Text
Halpern et al. "Aggregating Binary Judgments Ranked by Accuracy." AAAI Conference on Artificial Intelligence, 2021. doi:10.1609/AAAI.V35I6.16687Markdown
[Halpern et al. "Aggregating Binary Judgments Ranked by Accuracy." AAAI Conference on Artificial Intelligence, 2021.](https://mlanthology.org/aaai/2021/halpern2021aaai-aggregating/) doi:10.1609/AAAI.V35I6.16687BibTeX
@inproceedings{halpern2021aaai-aggregating,
title = {{Aggregating Binary Judgments Ranked by Accuracy}},
author = {Halpern, Daniel and Kehne, Gregory and Peters, Dominik and Procaccia, Ariel D. and Shah, Nisarg and Skowron, Piotr},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2021},
pages = {5456-5463},
doi = {10.1609/AAAI.V35I6.16687},
url = {https://mlanthology.org/aaai/2021/halpern2021aaai-aggregating/}
}