Evaluating Large Language Models in a Complex Hidden Role Game
Quick Take
This study evaluates LLMs like Llama 3.1 70B in the game Secret Hitler, revealing a significant gap in strategic depth versus rule-based agents, which align with human decisions 86.7% of the time. LLMs exhibited poor deception retention and shorter game durations, indicating limitations in complex manipulation capabilities.
Key Points
- Introduced metrics: Role Identification Accuracy, Deception Retention Rate, Game State Impact Rate.
- Llama 3.1 70B achieved only 59.7% accuracy in strategic decision-making.
- Fascist roles experienced up to 23.2% worse win rates with no improvement from reasoning techniques.
- Rule-based agents aligned with human voting decisions 86.7% of the time.
- Current LLM architectures struggle with complex, multi-turn manipulation.
Article Content
From source RSS / original summaryarXiv:2605. 22826v1 Announce Type: new Abstract: Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments. This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler. I introduce an open-source framework and novel metrics to measure performance: Role Identification Accuracy, Deception Retention Rate, and Game State Impact Rate.
By benchmarking models against rule-based algorithms and human games, I identify a gap between conversational ability and strategic depth. The study also analyzes the impact of reasoning-enhancement techniques on win rates and strategic reasoning. Neither Chain-of-Thought prompting nor internal memory bring improvements in performance, with up to 23. 2% worse win rates for fascist roles. While rule-based agents align with expert human voting decisions 86. 7% of the time, models like Llama 3.
1 70B achieve only a 59. 7% accuracy. Models playing as Fascists consistently yield negative impact scores and fail to sustain deception, resulting in roughly 40% shorter games compared to humans. These findings suggest that current architectures remain ineffective at complex, multi-turn manipulation. As capabilities advance, detecting when models begin to master these deceptive behaviors is crucial. The developed framework serves as a reproducible testbed for future alignment research.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.
