Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities
Quick Answer
The CARE framework evaluates LLMs' alignment with community attitudes, revealing a 'realism gap' in simulating authentic discourse.
Quick Take
Despite community prompts, like GPT-3 struggle to capture sociolinguistic dynamics effectively, indicating current alignment strategies are inadequate.
Key Points
- CARE benchmarks LLM discourse against real community responses to news events.
- The framework identifies a persistent 'realism gap' in LLM simulations.
- Explicit community prompts do not inherently improve LLM simulation fidelity.
- Divergent behavioral signatures among models suggest alignment strategies need improvement.
- Human-AI collaboration validates the framework's effectiveness in evaluating LLMs.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 27388v1 Announce Type: new Abstract: (LLMs) are increasingly utilized as proxies for computational social analysis; yet, their ability to faithfully represent the "thick descriptions" (Geertz, 1973) of human communities remains a critical challenge. Current evaluations often reduce social identity to static labels, sidelining how real-world groups navigate social shifts.
To bridge this gap, we introduce CARE (Community-Aware Reaction Evaluation), a reaction-centered framework that benchmarks LLM-simulated discourse against the authentic, event-contingent responses of distinct communities to real-world news.
By characterizing a fine-grained spectrum of illocutionary tones and the underlying attitudes they manifest--validated through human-AI collaboration--our diagnosis reveals a persistent "realism gap": steering LLMs with explicit community prompts fails to inherently improve simulation fidelity. Analysis further identifies divergent behavioral signatures among frontier models, suggesting that current alignment strategies remain insufficient for capturing the sociolinguistic dynamics of online groups.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.