Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities
Quick Answer
The CARE framework evaluates LLMs' alignment with community attitudes, revealing a 'realism gap' in simulating authentic discourse.
Quick Take
The CARE framework evaluates LLMs' alignment with community attitudes, revealing a 'realism gap' in simulating authentic discourse. Despite community prompts, LLMs like GPT-3 struggle to capture sociolinguistic dynamics effectively, indicating current alignment strategies are inadequate.
Key Points
- CARE benchmarks LLM discourse against real community responses to news events.
- The framework identifies a persistent 'realism gap' in LLM simulations.
- Explicit community prompts do not inherently improve LLM simulation fidelity.
- Divergent behavioral signatures among models suggest alignment strategies need improvement.
- Human-AI collaboration validates the framework's effectiveness in evaluating LLMs.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 27388v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly utilized as proxies for computational social analysis; yet, their ability to faithfully represent the "thick descriptions" (Geertz, 1973) of human communities remains a critical challenge. Current evaluations often reduce social identity to static labels, sidelining how real-world groups navigate social shifts.
To bridge this gap, we introduce CARE (Community-Aware Reaction Evaluation), a reaction-centered framework that benchmarks LLM-simulated discourse against the authentic, event-contingent responses of distinct communities to real-world news.
By characterizing a fine-grained spectrum of illocutionary tones and the underlying attitudes they manifest--validated through human-AI collaboration--our diagnosis reveals a persistent "realism gap": steering LLMs with explicit community prompts fails to inherently improve simulation fidelity. Analysis further identifies divergent behavioral signatures among frontier models, suggesting that current alignment strategies remain insufficient for capturing the sociolinguistic dynamics of online groups.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.