Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
Quick Answer
The paper critiques the reliability of large language models (LLMs) as measurement tools, emphasizing that agreement with human coders does not ensure construct validity.
Quick Take
The paper critiques the reliability of large language models (LLMs) as measurement tools, emphasizing that agreement with human coders does not ensure construct validity. It introduces 'grain calibration' to enhance validation by breaking down constructs and testing components against text, thus clarifying the measurement process.
Key Points
- Reliability of LLMs does not guarantee construct validity in measurement.
- Proposes 'grain calibration' to enhance validation of theoretical constructs.
- Decomposes constructs into clause-level components for precise testing.
- Validation focuses on the process rather than just the output.
- Clarifies whether errors stem from missed components or misidentified constructs.
Paper Resources
📖 Reader Mode
~2 min readAbstract:When a large language model (LLM) codes a construct in text as a human annotator would, that agreement makes the LLM a reliable coder. Yet reliability leaves construct validity untouched. The instrument may be theory-naive, reaching the code through a correlate that meets none of the demands the construct's theory makes, and no current method tells that apart from genuine measurement. We propose grain calibration as a method that closes the gap. It decomposes a construct into clause-level components, tests each against the text with extractive evidence, and combines the results through an explicit, theory-derived rule. Because the rule is stated rather than lodged in one opaque pass, its structure is evidence about the process rather than the output. It shows which components settled a code, and, when the code is wrong, whether a component was missed or an adjacent construct mistaken for it. Validation shifts from scoring an instrument's outputs against an annotator to showing that the instrument runs on the construct its theory specifies.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2606.28574 [cs.CL] |
| (or arXiv:2606.28574v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.28574 arXiv-issued DOI via DataCite |
Submission history
From: Manuel Pita PhD [view email]
[v1]
Fri, 26 Jun 2026 19:58:39 UTC (39 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.