Cluster and Predict Latents Patches for Improved Masked Image Modeling
Abstract
Masked Image Modeling (MIM) offers a promising approach to self-supervised representation learning, however existing MIM models still lag behind the state-of-the-art. In this paper, we systematically analyze target representations, loss functions, and architectures, to introduce CAPI -- a novel pure-MIM framework that relies on the prediction of latent clusterings. Our approach leverages a clustering-based loss, which is stable to train, and exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8\% accuracy on ImageNet and 32.1\% mIoU on ADE20K with simple linear probes, substantially outperforming previous MIM methods and approaching the performance of the current state-of-the-art, DINOv2.
Cite
Text
Darcet et al. "Cluster and Predict Latents Patches for Improved Masked Image Modeling." Transactions on Machine Learning Research, 2025.Markdown
[Darcet et al. "Cluster and Predict Latents Patches for Improved Masked Image Modeling." Transactions on Machine Learning Research, 2025.](https://mlanthology.org/tmlr/2025/darcet2025tmlr-cluster/)BibTeX
@article{darcet2025tmlr-cluster,
title = {{Cluster and Predict Latents Patches for Improved Masked Image Modeling}},
author = {Darcet, Timothée and Baldassarre, Federico and Oquab, Maxime and Mairal, Julien and Bojanowski, Piotr},
journal = {Transactions on Machine Learning Research},
year = {2025},
url = {https://mlanthology.org/tmlr/2025/darcet2025tmlr-cluster/}
}