Automating Geospatial Vision Tasks with a Large Language Model Agent

Abstract

Large Language Models (LLMs) have shown promise in automating code generation for data science tasks, yet they struggle with complex task sequences, especially in geospatial vision tasks. These difficulties stem from challenges in managing stepwise dependencies, aligning diverse data sources with spatial constraints, and accurately applying various geospatial libraries—often resulting in logical errors or hallucinations. To address these limitations, we introduce GeoAgent, an interactive framework designed to enable LLMs to automate geospatial vision tasks effectively. GeoAgent integrates a code interpreter, static analysis, and retrieval generation within a Monte Carlo Tree Search framework, creating a robust solution tailored to the geospatial data processing workflow. We introduce a new benchmark to evaluate GeoAgent’s performances on single- and multi-turn tasks, including geospatial data acquisition, analysis, and visualization across multiple Python libraries. Our experiments reveal that GeoAgent significantly outperforms baseline LLMs in function call accuracy, task pass rate and task completion, marking a substantial advancement in automating geospatial vision tasks and setting a new standard for LLM-driven geospatial data analysis.