GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models
Quick Answer
The GeoNatureAgent Benchmark introduces the first evaluation framework for LLM agents in environmental geospatial analysis, featuring 93 tasks across 18 categories.
Quick Take
Claude Sonnet 4 leads with 60.8% accuracy, while DeepSeek V3.2 offers 93% of its capability at 11x lower cost. The benchmark reveals significant limitations in reasoning for comparison tasks and highlights the need for structured against real APIs.
Key Points
- Benchmark includes 93 tasks across municipality analysis, spatial reasoning, and multilingual understanding.
- Claude Sonnet 4 achieves 60.8% accuracy, followed by DeepSeek V3.2 at 56.3%.
- DeepSeek V3.2 costs $0.011 per case, significantly lower than Claude Sonnet 4.
- Comparison tasks show 0% success rate, indicating reasoning limitations.
- Benchmark and API are publicly available for further research and development.
Paper Resources
Source Excerpt
arXiv:2606. 12821v1 Announce Type: new Abstract: Environmental scientists spend disproportionate effort on data wrangling rather than analysis, and AI agents that automate geospatial workflows remain unvalidated: no benchmark evaluates agents operating through structured against real APIs. We introduce the GeoNatureAgent Benchmark, the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.