Skill-Augmented AI Agents for Medical Research Analysis: An Exploratory Multi-Model Human Evaluation in an NSCLC Transcriptomic Biomarker Task
Quick Answer
This paper shows that An exploratory study evaluated skill-augmented AI agents, specifically OpenClaw, against native AI in analyzing NSCLC transcriptomic biomarkers.
Quick Take
Results indicated a slight quality improvement in skill-augmented outputs (mean 5.50) over native AI (mean 5.11), but the findings warrant further investigation due to limited expert agreement and variability.
Key Points
- Skill-augmented outputs scored higher on overall quality than native AI outputs.
- Expert reviewers rated skill-augmented outputs with a mean score of 5.50.
- Non-expert reviewers also favored skill-augmented outputs with a mean score of 4.72.
- Expert agreement was low, indicating variability in evaluations.
- Further research is needed to confirm findings and improve reliability.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 11830v1 Announce Type: new Abstract: Background. and AI agents are increasingly used to support biomedical research, but native model outputs may omit key analytical steps, misuse methods, or overstate conclusions. We evaluated whether autonomous access to a medical research skill package was associated with higher-quality AI-generated transcriptomic research-analysis outputs compared with native AI without skills. Methods.
We conducted an exploratory multi-model human evaluation using a non-small cell lung cancer immunotherapy biomarker task. Six model backbones were tested. The evaluation included 21 anonymized outputs: 9 native-AI outputs and 12 skill-augmented outputs generated through an AI agent implementation represented by OpenClaw. Four non-expert biomedical reviewers and two blinded experts evaluated each output, with two ratings from each reviewer type. The primary outcome was expert-rated overall quality. Results.
Skill-augmented outputs showed directionally higher expert overall quality than native-AI outputs (mean 5. 50 vs 5. 11; difference=0. 39; bootstrap 95\% CI, -0. 04 to 0. 90; Welch p=0. 156). Non-expert reviewer quality showed the same direction (mean 4. 72 vs 4. 47; difference=0. 26; bootstrap 95\% CI, -0. 25 to 0. 80; Welch p=0. 373). Expert agreement was limited (single-rating ICC=-0. 15), and model-specific effects were descriptive and heterogeneous. Conclusions.
Autonomous skill access showed a directional quality signal in this exploratory sample, but the signal was smaller than expert-rating noise and should not be interpreted as confirmatory evidence. The findings primarily motivate larger evaluations of skill-augmented AI agents with stronger reliability controls, platform replication, and biological-validity assessment.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.