Bipartite Ranking from Multiple Labels: On Loss Versus Label Aggregation
Abstract
Bipartite ranking is a fundamental supervised learning problem, with the goal of learning a ranking over instances with maximal area under the ROC curve (AUC) against a single binary target label. However, one may often observe multiple binary target labels, e.g., from distinct human annotators. How can one synthesize such labels into a single coherent ranking? In this work, we formally analyze two approaches to this problem—loss aggregation and label aggregation—by characterizing their Bayes-optimal solutions. We show that while both approaches can yield Pareto-optimal solutions, loss aggregation can exhibit label dictatorship: one can inadvertently (and undesirably) favor one label over others. This suggests that label aggregation can be preferable to loss aggregation, which we empirically verify.
Cite
Text
Lukasik et al. "Bipartite Ranking from Multiple Labels: On Loss Versus Label Aggregation." Proceedings of the 42nd International Conference on Machine Learning, 2025.Markdown
[Lukasik et al. "Bipartite Ranking from Multiple Labels: On Loss Versus Label Aggregation." Proceedings of the 42nd International Conference on Machine Learning, 2025.](https://mlanthology.org/icml/2025/lukasik2025icml-bipartite/)BibTeX
@inproceedings{lukasik2025icml-bipartite,
title = {{Bipartite Ranking from Multiple Labels: On Loss Versus Label Aggregation}},
author = {Lukasik, Michal and Chen, Lin and Narasimhan, Harikrishna and Menon, Aditya Krishna and Jitkrittum, Wittawat and Yu, Felix X. and Reddi, Sashank J. and Fu, Gang and Bateni, Mohammadhossein and Kumar, Sanjiv},
booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
year = {2025},
pages = {41074-41102},
volume = {267},
url = {https://mlanthology.org/icml/2025/lukasik2025icml-bipartite/}
}