ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?
Quick Answer
ArcANE introduces a novel benchmark for Role-Playing Language Agents (RPLAs) that evaluates character alignment with psychological arcs across 17 novels and 80 characters.
Quick Take
ArcANE introduces a novel benchmark for Role-Playing Language Agents (RPLAs) that evaluates character alignment with psychological arcs across 17 novels and 80 characters. Conditioning on Character Arcs significantly outperforms other context strategies, especially in scenarios not covered by the source text, enhancing model performance in open-weight models like ArcANE-8B/32B.
Key Points
- ArcANE benchmark spans 17 novels and 80 principal characters.
- Character Arc segments narratives into psychological phases for evaluation.
- Conditioning on Character Arc outperforms other context strategies.
- Performance gap is largest in scenarios outside the source text.
- Open-weight models fine-tuned on ArcANE data show improved results.
Article Excerpt
From source RSS / original summaryarXiv:2606. 05553v1 Announce Type: new Abstract: Role-playing language agents (RPLAs) should play characters whose values and behavior evolve as the story progresses, not maintain a fixed persona. Existing benchmarks measure factual recall at a given chapter, not whether responses align with the character's psychological trajectory, especially in scenarios the source text never explores.
We introduce ArcANE (Arc-Aware Narrative Evaluation), an automatically constructed benchmark spanning 17 novels and 80 principal characters. A Character Arc segments the narrative into phases along a psychological axis, and each probe poses the same scenario across phases, spanning both situations within the source text and situations beyond it.
Across six models and six context modes, conditioning on the Character Arc tops every other context strategy on every model, and the gap is largest on scenarios outside the source text where retrieval has nothing to find. We further fine-tune open-weight models on the same data to obtain ArcANE-8B/32B, which widen the Arc advantage even more on scenarios outside the source text.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.