A Delay Metric for Video Object Detection: What Average Precision Fails to Tell
Abstract
Average precision (AP) is a widely used metric to evaluate detection accuracy of image and video object detectors. In this paper, we analyze the object detection from video and point out that mAP alone is not sufficient to capture the temporal nature of video object detection. To tackle this problem, we propose a comprehensive metric, Average Delay (AD), to measure and compare detection delay. To facilitate delay evaluation, we carefully select a subset of ImageNet VID, which we name as ImageNet VIDT with an emphasis on complex trajectories. By extensively evaluating a wide range of detectors on VIDT, we show that most methods drastically increase the detection delay but still preserve mAP well. In other words, mAP is not sensitive enough to reflect the temporal characteristics of a video object detector. Our results suggest that video object detection methods should be evaluated with a delay metric, particularly for latency-critical applications such as autonomous vehicle perception.
Cite
Text
Mao et al. "A Delay Metric for Video Object Detection: What Average Precision Fails to Tell." Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. doi:10.1109/ICCV.2019.00066Markdown
[Mao et al. "A Delay Metric for Video Object Detection: What Average Precision Fails to Tell." Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.](https://mlanthology.org/iccv/2019/mao2019iccv-delay/) doi:10.1109/ICCV.2019.00066BibTeX
@inproceedings{mao2019iccv-delay,
title = {{A Delay Metric for Video Object Detection: What Average Precision Fails to Tell}},
author = {Mao, Huizi and Yang, Xiaodong and Dally, William J.},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
year = {2019},
doi = {10.1109/ICCV.2019.00066},
url = {https://mlanthology.org/iccv/2019/mao2019iccv-delay/}
}