Frontier LLM-based agents can overcome the ontology curation bottleneck for natural phenotypes
Quick Answer
This paper shows that Frontier LLMs from Anthropic and OpenAI effectively bridge the ontology curation gap for phenotype annotation, achieving inter-curator variability levels similar to trained human biocurators.
Quick Take
Frontier LLMs from Anthropic and OpenAI effectively bridge the ontology curation gap for phenotype annotation, achieving inter-curator variability levels similar to trained human biocurators. These agents significantly outperformed the Semantic CharaParser across all metrics, indicating a promising shift towards scalable automated solutions in comparative morphology.
Key Points
- Five frontier LLMs were evaluated against a Gold Standard of EQ annotations.
- Agents matched inter-curator variability of trained human biocurators.
- Best-performing agents approached but did not surpass top human curator results.
- LLMs outperformed Semantic CharaParser on all four evaluation metrics.
- This advancement addresses the scalability challenge in phenotype annotation.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 28965v1 Announce Type: new Abstract: Linking free-text phenotype descriptions to ontology terms, typically referred to as phenotype annotation, is essential for the cross-study integration of comparative morphological data. This labor intensive process has heavily relied on highly trained human experts, which makes it challenging to scale and thus a key bottleneck. Dahdul et al.
(2018) established a Gold Standard (GS) of Entity-Quality (EQ) annotations across seven phylogenetic studies and used it to evaluate three human curators and the Semantic CharaParser NLP tool with ontology-based semantic similarity metrics; they reported that machine-human consistency was significantly lower than inter-curator (human-human) consistency.
Here we revisit that benchmark with five frontier hosted LLMs from Anthropic and OpenAI, each operating as an "agentic curator" within a self-contained workspace that supplies the source publication PDF, the same annotation guide used by the original human curators, the four project ontologies (UBERON, PATO, BSPO, GO), and a validation script.
Evaluated against the same Gold Standard, every agent fell within the range of inter-curator variability of the three trained human biocurators of the original study; the best performing agents approached but did not reach the best performing human curator. Agents substantially outperformed Semantic CharaParser on all four metrics.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.