Evaluation design conditions the expert-vs-auto MeSH gap: a controlled comparison of bag-of-words and BiomedBERT on the Cohen benchmark
Quick Answer
This study compares bag-of-words and BiomedBERT classifiers using the Cohen benchmark, revealing that evaluation design significantly impacts the expert-vs-auto MeSH gap.
Quick Take
The bag-of-words model shows a gap of +0.096 WSS@95% under canonical evaluation, which narrows to +0.021 with 10-fold cross-validation, while BiomedBERT's performance is comparable at +0.020. The findings suggest that classifier performance can vary considerably based on evaluation methods.
Key Points
- Bag-of-words classifier shows a +0.096 WSS@95% gap in canonical evaluation.
- 10-fold cross-validation reduces the bag-of-words gap to +0.021.
- BiomedBERT's performance is comparable at +0.020 WSS@95%.
- Representation asymmetry affects results with 15.1% of inputs exceeding BiomedBERT's token limit.
- Evaluation design significantly alters conclusions about feature sources.
Paper Resources
Source Excerpt
A systematic review begins with someone reading thousands of abstracts to identify the few that are relevant, and classifiers are used to prioritise that reading. Their inputs are often augmented with Medical Subject Headings (MeSH), assigned either by expert indexers weeks or months after publication or by automatic tools at once. To our knowledge the two have not been compared directly as classifier features, and no previous work has asked whether that comparison's outcome depends on how the c
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.