BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos

Jennifer J. Sun, Lili Karashchuk, Amil Dravid, Serim Ryou, Sonia Fereidooni, John C. Tuthill, Aggelos Katsaggelos, Bingni W. Brunton, Georgia Gkioxari, Ann Kennedy, Yisong Yue, Pietro Perona

CVPR 2023 pp. 9001-9010

doi:10.1109/CVPR52729.2023.00869 /cvpr/2023/sun2023cvpr-bkind3d/

Abstract

Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised keypoint discovery is a promising strategy for estimating 3D poses without annotations. However, current keypoint discovery approaches commonly process single 2D views and do not operate in the 3D space. We propose a new method to perform self-supervised keypoint discovery in 3D from multi-view videos of behaving agents, without any keypoint or bounding box supervision in 2D or 3D. Our method, BKinD-3D, uses an encoder-decoder architecture with a 3D volumetric heatmap, trained to reconstruct spatiotemporal differences across multiple views, in addition to joint length constraints on a learned 3D skeleton of the subject. In this way, we discover keypoints without requiring manual supervision in videos of humans and rats, demonstrating the potential of 3D keypoint discovery for studying behavior.

PDF CVPR Semantic Scholar

Cite

Text

Sun et al. "BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos." Conference on Computer Vision and Pattern Recognition, 2023. doi:10.1109/CVPR52729.2023.00869

Markdown

[Sun et al. "BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos." Conference on Computer Vision and Pattern Recognition, 2023.](https://mlanthology.org/cvpr/2023/sun2023cvpr-bkind3d/) doi:10.1109/CVPR52729.2023.00869

BibTeX

@inproceedings{sun2023cvpr-bkind3d,
  title     = {{BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos}},
  author    = {Sun, Jennifer J. and Karashchuk, Lili and Dravid, Amil and Ryou, Serim and Fereidooni, Sonia and Tuthill, John C. and Katsaggelos, Aggelos and Brunton, Bingni W. and Gkioxari, Georgia and Kennedy, Ann and Yue, Yisong and Perona, Pietro},
  booktitle = {Conference on Computer Vision and Pattern Recognition},
  year      = {2023},
  pages     = {9001-9010},
  doi       = {10.1109/CVPR52729.2023.00869},
  url       = {https://mlanthology.org/cvpr/2023/sun2023cvpr-bkind3d/}
}