Can Multi-Agent LLMs Identify Their Peers? Stylometric Fingerprinting in Role-Constrained Political Analysis
Quick Answer
This study reveals that multi-agent LLMs, including Claude Sonnet 4.6 and Llama-3.3-70B, cannot effectively anonymize model identity in political analysis, as demonstrated by T5-base achieving a Macro F1 score of 0.991 under a new statement-disjoint cross-validation protocol.
Quick Take
The findings highlight significant implications for compliance with the EU AI Act and quality-critical deployments.
Key Points
- T5-base achieved a Macro F1 score of 0.991 under statement-disjoint cross-validation.
- Anonymization fails to neutralize model identity signals in political analysis.
- Study evaluates three classifiers: Claude Sonnet 4.6, Llama-3.3-70B, and fine-tuned T5-base.
- Findings impact compliance with EU AI Act Articles 13, 14, and 26.
- Performance knee identified at 40% of training data (~440 texts).
Paper Resources
Source Excerpt
arXiv:2606. 09854v1 Announce Type: new Abstract: (LLM) pipelines for political statement analysis are vulnerable to peer-preservation bias: models tend to protect peer models from deactivation and show identity-dependent scoring distortions. Prompt-level anonymization was proposed as a mitigation, but prior work simultaneously documented that stylometric fingerprints survive anonymization in role-constrained outputs - raising the question of whether this mitigation is sufficient.
This paper provides the first systematic investigation of whether LLMs can identify the model family behind political analysis texts under anonymization conditions. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.