Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Abstract
Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities.
Cite
Text
Li et al. "Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms." International Conference on Learning Representations, 2025.Markdown
[Li et al. "Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms." International Conference on Learning Representations, 2025.](https://mlanthology.org/iclr/2025/li2025iclr-ferretui/)BibTeX
@inproceedings{li2025iclr-ferretui,
title = {{Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms}},
author = {Li, Zhangheng and You, Keen and Zhang, Haotian and Feng, Di and Agrawal, Harsh and Li, Xiujun and Moorthy, Mohana Prasad Sathya and Nichols, Jeffrey and Yang, Yinfei and Gan, Zhe},
booktitle = {International Conference on Learning Representations},
year = {2025},
url = {https://mlanthology.org/iclr/2025/li2025iclr-ferretui/}
}