Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs
Abstract
Multimodal large language models (MLLMs) have shown remarkable capabilities across a broad range of tasks but their knowledge and abilities in the geographic and geospatial domains are yet to be explored, despite potential wide-ranging benefits to navigation, environmental research, and disaster response. We conduct a series of experiments exploring various vision capabilities of MLLMs within these domains, particularly focusing on the frontier model GPT-4V, and benchmark its performance against open-source counterparts. Our methodology involves challenging these models with a small-scale geographic benchmark consisting of a suite of visual tasks, testing their abilities across a spectrum of complexity. The analysis uncovers not only where such models excel, including instances where they outperform humans, but also where they falter, providing a balanced view of their capabilities in the geographic domain. To enable the comparison and evaluation of future models, we publicly release our benchmark.1
Cite
Text
Roberts et al. "Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2024. doi:10.1109/CVPRW63382.2024.00060Markdown
[Roberts et al. "Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2024.](https://mlanthology.org/cvprw/2024/roberts2024cvprw-charting/) doi:10.1109/CVPRW63382.2024.00060BibTeX
@inproceedings{roberts2024cvprw-charting,
title = {{Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs}},
author = {Roberts, Jonathan and Lüddecke, Timo and Sheikh, Rehan and Han, Kai and Albanie, Samuel},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
year = {2024},
pages = {554-563},
doi = {10.1109/CVPRW63382.2024.00060},
url = {https://mlanthology.org/cvprw/2024/roberts2024cvprw-charting/}
}