Revisiting Multimodal Positional Encoding in Vision–Language Models
Abstract
Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors—ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding. Code is avaliable at https://github.com/JJJYmmm/Multimodal-RoPEs.
Cite
Text
Huang et al. "Revisiting Multimodal Positional Encoding in Vision–Language Models." International Conference on Learning Representations, 2026.Markdown
[Huang et al. "Revisiting Multimodal Positional Encoding in Vision–Language Models." International Conference on Learning Representations, 2026.](https://mlanthology.org/iclr/2026/huang2026iclr-revisiting/)BibTeX
@inproceedings{huang2026iclr-revisiting,
title = {{Revisiting Multimodal Positional Encoding in Vision–Language Models}},
author = {Huang, Jie and Liu, Xuejing and Song, Sibo and Hou, RuiBing and Chang, Hong and Lin, Junyang and Bai, Shuai},
booktitle = {International Conference on Learning Representations},
year = {2026},
url = {https://mlanthology.org/iclr/2026/huang2026iclr-revisiting/}
}