Prompt Framing Distorts Count-Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring
Quick Answer
This study reveals that count-based F1 scores can inflate without genuine improvements in error detection, highlighting a significant gap termed F1 Inflation.
Quick Take
Using ErrorBench, it was found that anchored prompts can inflate F1 scores by up to 0.79 points, suggesting that evaluations should prioritize span-aware metrics over pre-populated error counts.
Key Points
- Count-based F1 scores can inflate significantly without improving span localization.
- ErrorBench evaluated six LLMs, revealing up to 0.79 points of F1 Inflation.
- Blind-to-Anchored prompt shift raised Count-F1 by +0.21 on average.
- GPT/Claude systems produced larger count responses under stress testing.
- LLM evaluations should report span-aware metrics alongside count-based metrics.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Count-based F1 is widely used as a proxy for LLM error-detection quality, but this paper shows that it can rise dramatically without a corresponding improvement in span localization, a gap termed F1 Inflation. The paper introduces ErrorBench, a controlled stress-test protocol for prompt-induced count distortion. ErrorBench evaluates six contemporary LLMs under five prompt conditions over 4,290 responses from 143 CoNLL-2014 passages. Under CoNLL-2014 M2-style scoring, anchored prompts produce up to 0.79 points of F1 Inflation, and up to 0.96 under strict matching. A 100-passage replication using the official ERRANT 3.0.0 pipeline and multi-reference scoring reproduces the pattern: averaged over six models, the Blind-to-Anchored prompt shift raises Count-F1 by +0.21 while raising multi-reference ERRANT F0.5 by only +0.04. The study finds larger count responses in highly instruction-compliant GPT/Claude systems and smaller responses in the Gemini family under this stress-test protocol. The findings suggest that LLM proofreading and document-review evaluations should avoid pre-populated error counts and should report span-aware metrics alongside count-based metrics.
| Comments: | 15 pages, 6 figures, 12 tables. Preprint under review |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; I.2.6; H.3.3 |
| Cite as: | arXiv:2607.01240 [cs.CL] |
| (or arXiv:2607.01240v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01240 arXiv-issued DOI via DataCite |
Submission history
From: Dekun Yang [view email]
[v1]
Sun, 3 May 2026 12:21:42 UTC (83 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.