When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding
Quick Answer
The study reveals that while clearer expert codebooks enhance classification performance in political event coding, they do not guarantee behavioral reliability in LLMs.
Quick Take
This indicates that should be evaluated not just on accuracy but also on their ability to maintain the coding logic essential for social-science research.
Key Points
- Clearer codebooks significantly improve classification performance for fine-grained political event coding.
- LLMs may achieve high accuracy but fail behavioral reliability tests with controlled codebook changes.
- Expert-written codebooks are crucial for transforming text into structured data in social sciences.
- Behavioral reliability is essential for ensuring meaningful coded outputs in research.
- The study emphasizes the need for comprehensive evaluation criteria beyond just accuracy.
Paper Resources
Source Excerpt
arXiv:2606. 06781v1 Announce Type: new Abstract: High accuracy does not necessarily make an a faithful coder. This issue matters because many social-science studies rely on expert-written codebooks to turn text into structured data. We study this problem in political event coding, a challenging source-target relation classification task beyond ordinary sentence-level classification, where models must determine what one actor did to another using detailed coding rules.
We test whether expert codebooks become more effective when operationalized into LLM-friendly forms with clearer definitions, examples, retrieved context, and rules for difficult cases. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.