When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis
Quick Take
This study critiques the evaluation of large language models (LLMs) in public comment analysis, revealing that thematic divergence among four LLMs exceeds variations within a single model. An Interpretive Audit Pipeline is proposed to address interpretive complexity, highlighting the need for disagreement-based evaluation alongside traditional accuracy metrics.
Key Points
- Four LLMs analyzed public comments on a USDA docket, revealing significant thematic divergence.
- Standard accuracy metrics fail to capture meaningful differences in model categorizations.
- Human annotators introduced unique framings absent in LLM outputs during revisions.
- The proposed Interpretive Audit Pipeline aids in identifying genuinely ambiguous public input.
- Disagreement-based evaluation complements traditional metrics for LLM-assisted coding.
Article Content
From source RSS / original summaryarXiv:2605. 29025v1 Announce Type: new Abstract: Federal agencies are deploying large language models (LLMs) to categorize public comment corpora, where the model's organization of the record shapes what policymakers see and which arguments register. Standard evaluation, anchored on stance accuracy against a small validated set, cannot detect when different models produce materially different categorizations of the same public input.
We propose an Interpretive Audit Pipeline that treats multi-model disagreement as diagnostic of interpretive complexity and directs human review toward genuinely ambiguous public input. Analyzing 1,260 public comments on a federal USDA docket across four LLMs, we find that inter-model thematic divergence exceeds within-model prompt variation, and that an expert rubric suppresses deep interpretive disagreement without resolving it.
In a two-stage labeling study on a stratified 40-comment subsample, four LLMs and a human annotator labeled independently and then revised after seeing the others' labels. Revision behavior varied across labelers, and the human annotator's revisions frequently introduced framings absent from the ensemble's collective output. We argue disagreement-based evaluation is a necessary complement to accuracy metrics for LLM-assisted interpretive coding.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
The Redpanda Agentic Data Plane (ADP) introduces out-of-band metadata channels to enhance the safety of autonomous AI agents, ensuring secure data access and tamper-proof audit trails. This architecture mitigates risks associated with unpredictable AI behavior by enforcing governance throughout the agent lifecycle, demonstrated in a multi-agent trading system with strict data scoping and approval thresholds.