Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity
Quick Answer
A study compares LLMs and fine-tuned RoBERTa for extracting circumstances from NVDRS narratives, revealing LLMs excel in low-prevalence scenarios.
Quick Take
The hybrid model adapts prompt strategies based on complexity, with GPT-5.2, Gemini 2.5 Pro, and Llama-3 70B showing consistent performance across complex circumstances.
Key Points
- outperform fine-tuned models in low-prevalence circumstances with insufficient training data.
- A 'Complexity Score' algorithm predicts when detailed prompts enhance performance.
- Hybrid approach selects prompt strategies based on the complexity of circumstances.
- Study evaluated 25 inferentially complex circumstances from NVDRS.
- Findings support using LLMs for rare, complex cases and fine-tuned models for common ones.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Suicide is a leading cause of death in the United States, and understanding the circumstances that precede it requires extracting structured information from death investigation narratives. Many of these circumstances require semantic inference beyond simple keyword matching. We develop a ``Complexity Score'' algorithm that analyzes coding manual structure to predict when detailed prompts with full coding guidelines improve over name-only prompts. We then construct a hybrid approach that selects prompt strategy per circumstance. We evaluate large language models (LLMs) against fine-tuned RoBERTa on 25 inferentially complex circumstances from the National Violent Death Reporting System (NVDRS). We found that LLMs substantially outperform on low-prevalence circumstances where training data is insufficient. We further demonstrate that our framework generalizes across frontier LLMs, with GPT-5.2, Gemini 2.5 Pro and Llama-3 70B showing consistent performance patterns. These findings support a hybrid architecture where LLMs handle rare, inferentially complex circumstances while fine-tuned models handle common ones.
| Comments: | Accepted at IEEE ICHI 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.21845 [cs.CL] |
| (or arXiv:2605.21845v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21845 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Geoffrey Martin [view email]
[v1]
Thu, 21 May 2026 00:33:52 UTC (20 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.