Seduced by the Narrative: Assessing Rule Adherence in Semi-Open Textual Sandboxes
Quick Answer
The study introduces CoC-Seduce, a benchmark assessing rule adherence in LLMs like GPT-5.4 and Claude Sonnet 4.6 in semi-open environments.
Quick Take
The study introduces CoC-Seduce, a benchmark assessing rule adherence in LLMs like GPT-5.4 and Claude Sonnet 4.6 in semi-open environments. It reveals that adversarial attacks using pseudo-logical reasoning undermine adjudication robustness, exposing knowledge gaps across various models and settings.
Key Points
- CoC-Seduce benchmark tests 20 LLMs against 5,376 adversarial samples.
- Pseudo-logic is the primary attack vector identified in the study.
- Robustness of models does not correlate with scale or reasoning mechanisms.
- Cross-cultural settings reveal systematic knowledge gaps in evaluated models.
- The benchmark is based on Tabletop Role-Playing Game mechanics.
Paper Resources
📖 Reader Mode
~2 min readAbstract:As LLMs are increasingly deployed as autonomous adjudicators in semi-open textual game environments, robust rule adherence becomes critical when user intent conflicts with system rules. However, these models are trained to be helpful and compliant, leaving them vulnerable to a class of attacks we term \textit{Rhetorical Injection}, where adversarial users exploit narrative framing techniques such as pseudo-logical reasoning and authoritative coercion to bypass adjudication logic. We present CoC-Seduce, a multi-agent adversarial benchmark built on Tabletop Role-Playing Game (TRPG) mechanics, an ideal instantiation of semi-open environments where rules are explicit for adjudication, yet interaction remains entirely in natural language. Three frontier models, i.e., GPT-5.4, Claude Sonnet 4.6, Gemini 3.5 Flash, serve as adversarial generators producing 5,376 samples across 4 world settings and 16 skill categories. We then benchmark 20 target adjudicators against this corpus. Evaluation across 20 models reveals that neither model scale nor explicit reasoning mechanisms reliably confer adjudication robustness, with \textsc{Pseudo-Logic} emerging as the dominant attack vector and cross-cultural settings exposing systematic knowledge gaps across all evaluated families. Project page: this https URL
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.02802 [cs.CL] |
| (or arXiv:2607.02802v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02802 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Weiying Chen [view email]
[v1]
Thu, 2 Jul 2026 22:25:35 UTC (365 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.