On the Rankability of Visual Embeddings

Abstract

We study whether visual embedding models capture continuous, ordinal attributes along linear directions, which we term _rank axes_. We define a model as _rankable_ for an attribute if projecting embeddings onto such an axis preserves the attribute's order. Across 7 popular encoders and 9 datasets with attributes like age, crowd count, head pose, aesthetics, and recency, we find that many embeddings are inherently rankable. Surprisingly, a small number of samples, or even just two extreme examples, often suffice to recover meaningful rank axes, without full-scale supervision. These findings open up new use cases for image ranking in vector databases and motivate further study into the structure and learning of rankable embeddings.

PDF NeurIPS OpenReview Semantic Scholar

Cite

Text

Sonthalia et al. "On the Rankability of Visual Embeddings." Advances in Neural Information Processing Systems, 2025.

Markdown

[Sonthalia et al. "On the Rankability of Visual Embeddings." Advances in Neural Information Processing Systems, 2025.](https://mlanthology.org/neurips/2025/sonthalia2025neurips-rankability/)

BibTeX

@inproceedings{sonthalia2025neurips-rankability,
  title     = {{On the Rankability of Visual Embeddings}},
  author    = {Sonthalia, Ankit and Uselis, Arnas and Oh, Seong Joon},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2025},
  url       = {https://mlanthology.org/neurips/2025/sonthalia2025neurips-rankability/}
}