OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images
Abstract
Enhancing the robustness of vision algorithms in real-world scenarios is challenging. One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or ignore the effects of individual nuisance factors. We introduce ROBIN, a benchmark dataset that includes out-of-distribution examples of 10 object categories in terms of pose, shape, texture, context and the weather conditions, and enables benchmarking models for image classification, object detection, and 3D pose estimation. Our experiments using popular baseline methods reveal that: 1) Some nuisance factors have a much stronger negative effect on the performance compared to others, also depending on the vision task. 2) Current approaches to enhance robustness have only marginal effects, and can even reduce robustness. 3) We do not observe significant differences between convolutional and transformer architectures. We believe our dataset provides a rich testbed to study robustness and will help push forward research in this area.
Cite
Text
Zhao et al. "OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images." Proceedings of the European Conference on Computer Vision (ECCV), 2022. doi:10.1007/978-3-031-20074-8_10Markdown
[Zhao et al. "OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images." Proceedings of the European Conference on Computer Vision (ECCV), 2022.](https://mlanthology.org/eccv/2022/zhao2022eccv-oodcv/) doi:10.1007/978-3-031-20074-8_10BibTeX
@inproceedings{zhao2022eccv-oodcv,
title = {{OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images}},
author = {Zhao, Bingchen and Yu, Shaozuo and Ma, Wufei and Yu, Mingxin and Mei, Shenxiao and Wang, Angtian and He, Ju and Yuille, Alan and Kortylewski, Adam},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2022},
doi = {10.1007/978-3-031-20074-8_10},
url = {https://mlanthology.org/eccv/2022/zhao2022eccv-oodcv/}
}