Improving Learning-from-Crowds Through Expert Validation
Abstract
Although several effective learning-from-crowd methods have been developed to infer correct labels from noisy crowdsourced labels, a method for post-processed expert validation is still needed. This paper introduces a semi-supervised learning algorithm that is capable of selecting the most informative instances and maximizing the influence of expert labels. Specifically, we have developed a complete uncertainty assessment to facilitate the selection of the most informative instances. The expert labels are then propagated to similar instances via regularized Bayesian inference. Experiments on both real-world and simulated datasets indicate that given a specific accuracy goal (e.g., 95%) our method reduces expert effort from 39% to 60% compared with the state-of-the-art method.
Cite
Text
Liu et al. "Improving Learning-from-Crowds Through Expert Validation." International Joint Conference on Artificial Intelligence, 2017. doi:10.24963/IJCAI.2017/324Markdown
[Liu et al. "Improving Learning-from-Crowds Through Expert Validation." International Joint Conference on Artificial Intelligence, 2017.](https://mlanthology.org/ijcai/2017/liu2017ijcai-improving/) doi:10.24963/IJCAI.2017/324BibTeX
@inproceedings{liu2017ijcai-improving,
title = {{Improving Learning-from-Crowds Through Expert Validation}},
author = {Liu, Mengchen and Jiang, Liu and Liu, Junlin and Wang, Xiting and Zhu, Jun and Liu, Shixia},
booktitle = {International Joint Conference on Artificial Intelligence},
year = {2017},
pages = {2329-2336},
doi = {10.24963/IJCAI.2017/324},
url = {https://mlanthology.org/ijcai/2017/liu2017ijcai-improving/}
}