Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study
Quick Answer
The study introduces EnterpriseMem-Bench, a multi-turn Text-to-SQL benchmark revealing that stateless models collapse in accuracy by Turn 3.
Quick Take
The study introduces EnterpriseMem-Bench, a multi-turn Text-to-SQL benchmark revealing that stateless models collapse in accuracy by Turn 3. Evaluating models like GPT-5 and Claude Sonnet, findings show memory architecture complexity does not guarantee improved accuracy, with Claude Sonnet 4.6 underperforming its predecessor by up to 33 percentage points.
Key Points
- EnterpriseMem-Bench consists of 300 sessions and 1,400 turns across three domains.
- Stateless models show zero execution accuracy by Turn 3 in multi-turn settings.
- Memory architecture complexity yields variable effects, with working memory being crucial.
- Claude Sonnet 4.6 underperforms Sonnet 4.5 by 17-33 percentage points on SEC EDGAR.
- The study introduces the Memory Benefit Score (MBS) for per-turn evaluation.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 26394v1 Announce Type: new Abstract: Multi-turn Text-to-SQL is central to enterprise analytics yet remains predominantly evaluated in single-turn settings. We introduce EnterpriseMem-Bench, a multi-turn Text-to-SQL benchmark of 300 sessions and 1,400 turns built programmatically from three enterprise domains (BIRD financial, SEC EDGAR, Northwind), with deterministic ground truth and per-turn memory-critical annotation. We evaluate five frontier models -- GPT-5 mini, GPT-5. 2, Claude Sonnet 4.
5, Sonnet 4. 6, and Opus 4. 6 -- across five memory conditions enabling a three-way ablation isolating working-memory window size, episodic retrieval, and semantic augmentation as independent effects. All Claude models are evaluated with extended thinking enabled to maintain parity with GPT reasoning models. We introduce the Memory Benefit Score (MBS) as a per-turn diagnostic metric.
Four findings emerge: (1) stateless multi-turn Text-to-SQL collapses to zero execution accuracy by Turn 3 across all five models, even under reasoning; (2) memory-architecture complexity does not monotonically improve accuracy -- working memory dominates, and additional components produce model- and dataset-dependent effects from +14 to -16 percentage points; (3) Claude Sonnet 4. 6 underperforms Sonnet 4.
5 by 17-33pp on SEC EDGAR across conditions, a generational regression persisting under reasoning; (4) under reasoning, Claude error distributions become mono-modal -- every non-correct turn is a wrong-result error. We release the benchmark, agent, and evaluation code.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.