GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models
Quick Answer
The GeoNatureAgent Benchmark introduces the first evaluation framework for LLM agents in environmental geospatial analysis, featuring 93 tasks across 18 categories.
Quick Take
The GeoNatureAgent Benchmark introduces the first evaluation framework for LLM agents in environmental geospatial analysis, featuring 93 tasks across 18 categories. Claude Sonnet 4 leads with 60.8% accuracy, while DeepSeek V3.2 offers 93% of its capability at 11x lower cost. The benchmark reveals significant limitations in reasoning for comparison tasks and highlights the need for structured against real APIs.
Key Points
- Benchmark includes 93 tasks across municipality analysis, spatial reasoning, and multilingual understanding.
- Claude Sonnet 4 achieves 60.8% accuracy, followed by DeepSeek V3.2 at 56.3%.
- DeepSeek V3.2 costs $0.011 per case, significantly lower than Claude Sonnet 4.
- Comparison tasks show 0% success rate, indicating reasoning limitations.
- Benchmark and API are publicly available for further research and development.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 12821v1 Announce Type: new Abstract: Environmental scientists spend disproportionate effort on data wrangling rather than analysis, and AI agents that automate geospatial workflows remain unvalidated: no benchmark evaluates agents operating through structured against real APIs. We introduce the GeoNatureAgent Benchmark, the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API.
It comprises 93 tasks across 18 categories, covering municipality analysis, multi-turn conversation, spatial reasoning, cross-indicator synthesis, error handling and recovery, ranking, comparison, multilingual understanding, habitat analysis, and task rejection. Tasks are evaluated against an open, self-hostable API serving three environmental indicators across Spain and Portugal via sixteen tools. We evaluate seven LLMs (Claude Sonnet 4, DeepSeek V3. 2, GLM-5, Gemini 2.
5 Pro, Qwen3-235B, GPT-OSS-120B, Llama 4 Scout) under three temperature-1. 0 seeds, reporting capability and per-case cost as orthogonal axes. We find: (1) Claude Sonnet 4 leads at 60. 8% +/- 0. 8%, followed by DeepSeek V3. 2 at 56. 3% +/- 3. 1%, with no other model above 51%; (2) the cost-accuracy Pareto frontier is occupied mostly by open-weight models, with DeepSeek V3. 2 offering 93% of Claude's capability at 11x lower cost ($0.
011/case); (3) comparison tasks remain universally unsolved (0% on close-value comparisons), exposing systematic reasoning limits; and (4) structured tool calling against a real API is more discriminative than general-purpose GIS benchmarks, with accuracies 25-35 points lower. We further show extensibility by integrating BigEarthNet V2 land cover for Portugal alongside Spanish CO2 and erosion indicators. The benchmark, harness, and self-hostable API are publicly available.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Arbor: Tree Search as a Cognition Layer for Autonomous Agents
Arbor introduces a multi-agent framework utilizing structured tree search for optimizing LLM inference, achieving up to 193% throughput-latency improvement compared to vendor-optimized systems. It employs an Orchestrator and Critic agent for stability and coordination, demonstrating hardware-agnostic performance with minimal variance.