Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents
Quick Answer
Dialogue SWE-Bench introduces a benchmark for evaluating coding agents' dialogue capabilities in real-world software engineering tasks.
Quick Take
A novel user simulator and schema-guided agent improve dialogue performance by 3-14%, highlighting that coding proficiency doesn't equate to dialogue effectiveness.
Key Points
- Dialogue evaluates coding agents through user dialogue rather than autonomous tasks.
- A persona-grounded user simulator enhances task evaluation and dialogue quality assessment.
- The schema-guided agent outperforms strong baselines by 3-14% in dialogue tasks.
- Findings suggest dialogue capability is a distinct and underexplored aspect of coding agents.
Paper Resources
Source Excerpt
arXiv:2606. 13995v1 Announce Type: new Abstract: AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue , an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user.
We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.