GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models
Quick Answer
The GeoNatureAgent Benchmark introduces the first evaluation framework for LLM agents in environmental geospatial analysis, featuring 93 tasks across 18 categories.
Quick Take
Claude Sonnet 4 leads with 60.8% accuracy, while DeepSeek V3.2 offers 93% of its capability at 11x lower cost. The benchmark reveals significant limitations in reasoning for comparison tasks and highlights the need for structured against real APIs.
Key Points
- Benchmark includes 93 tasks across municipality analysis, spatial reasoning, and multilingual understanding.
- Claude Sonnet 4 achieves 60.8% accuracy, followed by DeepSeek V3.2 at 56.3%.
- DeepSeek V3.2 costs $0.011 per case, significantly lower than Claude Sonnet 4.
- Comparison tasks show 0% success rate, indicating reasoning limitations.
- Benchmark and API are publicly available for further research and development.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 12821v1 Announce Type: new Abstract: Environmental scientists spend disproportionate effort on data wrangling rather than analysis, and AI agents that automate geospatial workflows remain unvalidated: no benchmark evaluates agents operating through structured against real APIs. We introduce the GeoNatureAgent Benchmark, the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API.
It comprises 93 tasks across 18 categories, covering municipality analysis, multi-turn conversation, spatial reasoning, cross-indicator synthesis, error handling and recovery, ranking, comparison, multilingual understanding, habitat analysis, and task rejection. Tasks are evaluated against an open, self-hostable API serving three environmental indicators across Spain and Portugal via sixteen tools. We evaluate seven (Claude Sonnet 4, DeepSeek V3. 2, GLM-5, Gemini 2.
5 Pro, Qwen3-235B, GPT-OSS-120B, Llama 4 Scout) under three temperature-1. 0 seeds, reporting capability and per-case cost as orthogonal axes. We find: (1) Claude Sonnet 4 leads at 60. 8% +/- 0. 8%, followed by DeepSeek V3. 2 at 56. 3% +/- 3. 1%, with no other model above 51%; (2) the cost-accuracy Pareto frontier is occupied mostly by open-weight models, with DeepSeek V3. 2 offering 93% of Claude's capability at 11x lower cost ($0.
011/case); (3) comparison tasks remain universally unsolved (0% on close-value comparisons), exposing systematic reasoning limits; and (4) structured tool calling against a real API is more discriminative than general-purpose GIS benchmarks, with accuracies 25-35 points lower. We further show extensibility by integrating BigEarthNet V2 land cover for Portugal alongside Spanish CO2 and erosion indicators. The benchmark, harness, and self-hostable API are publicly available.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.