MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos with Spherical Buffers and Padded Convolutions

Mathias Parger, Chengcheng Tang, Thomas Neff, Christopher D. Twigg, Cem Keskin, Robert Wang, Markus Steinberger

ICCV 2023 pp. 17292-17301

doi:10.1109/ICCV51070.2023.01586 /iccv/2023/parger2023iccv-motiondeltacnn/

Abstract

Convolutional neural network inference on video input is computationally expensive and requires high memory bandwidth. Recently, DeltaCNN managed to reduce the cost by only processing pixels with significant updates over the previous frame. However, DeltaCNN relies on static camera input. Moving cameras add new challenges in how to fuse newly unveiled image regions with already processed regions efficiently to minimize the update rate - without increasing memory overhead and without knowing the camera extrinsics of future frames. In this work, we propose MotionDeltaCNN, a sparse CNN inference framework that supports moving cameras. We introduce spherical buffers and padded convolutions to enable seamless fusion of newly unveiled regions and previously processed regions - without increasing memory footprint. Our evaluation shows that we outperform DeltaCNN by up to 90% for moving camera videos.

PDF ICCV Semantic Scholar

Cite

Text

Parger et al. "MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos with Spherical Buffers and Padded Convolutions." International Conference on Computer Vision, 2023. doi:10.1109/ICCV51070.2023.01586

Markdown

[Parger et al. "MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos with Spherical Buffers and Padded Convolutions." International Conference on Computer Vision, 2023.](https://mlanthology.org/iccv/2023/parger2023iccv-motiondeltacnn/) doi:10.1109/ICCV51070.2023.01586

BibTeX

@inproceedings{parger2023iccv-motiondeltacnn,
  title     = {{MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos with Spherical Buffers and Padded Convolutions}},
  author    = {Parger, Mathias and Tang, Chengcheng and Neff, Thomas and Twigg, Christopher D. and Keskin, Cem and Wang, Robert and Steinberger, Markus},
  booktitle = {International Conference on Computer Vision},
  year      = {2023},
  pages     = {17292-17301},
  doi       = {10.1109/ICCV51070.2023.01586},
  url       = {https://mlanthology.org/iccv/2023/parger2023iccv-motiondeltacnn/}
}