Difficulty Estimation with Action Scores for Computer Vision Tasks

Octavio Arriaga, Sebastian Palacio, Matias Valdenegro-Toro

CVPRW 2023 pp. 245-253

doi:10.1109/CVPRW59228.2023.00030 /cvprw/2023/arriaga2023cvprw-difficulty/

Abstract

As more machine learning models are now being applied in real world scenarios it has become crucial to evaluate their difficulties and biases. In this paper we present an unsupervised method for calculating a difficulty score based on the accumulated loss per epoch. Our proposed method does not require any modification to the model, neither any external supervision, and it can be easily applied to a wide range of machine learning tasks. We provide results for the tasks of image classification, image segmentation, and object detection. We compare our score against similar metrics and provide theoretical and empirical evidence of their difference. Furthermore, we show applications of our proposed score for detecting incorrect labels, and test for possible biases.

PDF CVPRW Semantic Scholar

Cite

Text

Arriaga et al. "Difficulty Estimation with Action Scores for Computer Vision Tasks." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2023. doi:10.1109/CVPRW59228.2023.00030

Markdown

[Arriaga et al. "Difficulty Estimation with Action Scores for Computer Vision Tasks." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2023.](https://mlanthology.org/cvprw/2023/arriaga2023cvprw-difficulty/) doi:10.1109/CVPRW59228.2023.00030

BibTeX

@inproceedings{arriaga2023cvprw-difficulty,
  title     = {{Difficulty Estimation with Action Scores for Computer Vision Tasks}},
  author    = {Arriaga, Octavio and Palacio, Sebastian and Valdenegro-Toro, Matias},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  year      = {2023},
  pages     = {245-253},
  doi       = {10.1109/CVPRW59228.2023.00030},
  url       = {https://mlanthology.org/cvprw/2023/arriaga2023cvprw-difficulty/}
}