NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama
Quick Answer
The NarrativeWorldBench introduces a benchmark for evaluating long-form audio drama models, revealing that closed-frontier systems plateau at F1 scores between 0.78 and 0.81.
Quick Take
The N-VSSM model outperforms competitors with F1 scores of 0.84 across various horizons while using 4x less compute, and it shows a 71% preference among professional authors for long-arc consistency.
Key Points
- Benchmark evaluates 21 models on narrative metrics across long horizons.
- Closed-frontier systems show a drop of -0.20 F1 at horizon h=200.
- N-VSSM maintains a structured latent state with 256 dimensions.
- Cultural Transfer Function improves cross-language fidelity by +0.20 to +0.23.
- N-VSSM preferred by 71% of authors for long-arc consistency.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 17391v1 Announce Type: new Abstract: Long-form serialized audio drama, with arcs that run for 200 to 800 episodes, is a major creative medium and a setting where frontier (LLMs) fail. We benchmark 21 models, spanning classical, fine-tuned, open-frontier, closed-frontier, and reasoning tiers, on a uniform set of structural narrative metrics. All closed-frontier systems saturate at a plot-beat F1 in the band [0. 78, 0. 81] and collapse by about -0. 20 F1 at horizon h=200.
We introduce NarrativeWorldBench, an open benchmark of nine narrative-structure metrics evaluated across horizons h in {10, 20, 50, 100, 200}, with cross-lingual evaluation across four Indic languages (Hindi, Tamil, Telugu, Marathi). We introduce N-VSSM, a Narrative Variational State-Space Model that maintains a structured 256-dimensional latent world state over more than 200 episodes via a Mamba-2 backbone with an event-conditioned posterior and an 8B decoder. N-VSSM holds plot-beat F1 >= 0.
84 across all horizons at 4x lower compute than the closed-frontier band. A learned Cultural Transfer Function lifts cross-language fidelity by +0. 20 to +0. 23 Likert points. In a within-subjects writer study (n = 12 professional authors, 240 trials), N-VSSM is preferred over Claude Opus 4. 5 on long-arc consistency 71% of the time and rated +1. 3 Likert points higher on controllability.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.