Inferring Hidden Statuses and Actions in Video by Causal Reasoning
Abstract
In the physical world, cause and effect are inseparable: ambient conditions trigger humans to perform actions, thereby driving status changes of objects. In video, these actions and statuses may be hidden due to ambiguity, occlusion, or because they are otherwise unobservable, but humans nevertheless perceive them. In this paper, we extend the Causal And-Or Graph (C-AOG) to a sequential model representing actions and their effects on objects over time, and we build a probability model for it. For inference, we apply a Viterbi algorithm, grounded on probabilistic detections from video, to fill in hidden and misdetected actions and statuses. We analyze our method on a new video dataset that showcases causes and effects. Our results demonstrate the effectiveness of reasoning with causality over time.
Cite
Text
Fire and Zhu. "Inferring Hidden Statuses and Actions in Video by Causal Reasoning." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2017. doi:10.1109/CVPRW.2017.13Markdown
[Fire and Zhu. "Inferring Hidden Statuses and Actions in Video by Causal Reasoning." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2017.](https://mlanthology.org/cvprw/2017/fire2017cvprw-inferring/) doi:10.1109/CVPRW.2017.13BibTeX
@inproceedings{fire2017cvprw-inferring,
title = {{Inferring Hidden Statuses and Actions in Video by Causal Reasoning}},
author = {Fire, Amy Sue and Zhu, Song-Chun},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
year = {2017},
pages = {48-56},
doi = {10.1109/CVPRW.2017.13},
url = {https://mlanthology.org/cvprw/2017/fire2017cvprw-inferring/}
}