CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Quick Answer
CausalDS introduces a novel benchmark for evaluating causal reasoning in data-science agents, integrating structural causal models with observational data and synthetic narratives.
Quick Take
CausalDS introduces a novel benchmark for evaluating causal reasoning in data-science agents, integrating structural causal models with observational data and synthetic narratives. This benchmark spans Pearl's causal reasoning rungs and includes tasks that require data science coding, uncertainty quantification, and , addressing the limitations of existing datasets.
Key Points
- CausalDS benchmarks causal reasoning in data-science workflows using structural causal models.
- Tasks include data science coding, uncertainty quantification, and recognizing unanswerable questions.
- Synthetic narratives are grounded in realistic domains, enhancing the benchmark's applicability.
- The benchmark aims to reduce risks associated with 'causal parroting' in AI models.
- It addresses gaps in existing benchmarks by combining causal reasoning with realistic data analysis.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. We optionally ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the "causal parrot" risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs, with typical data-science prediction tasks appearing as Rung 1. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first-class scored outcome. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.
| Comments: | 55 pages, 10 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.08093 [cs.AI] |
| (or arXiv:2607.08093v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.08093 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Andrej Leban [view email]
[v1]
Thu, 9 Jul 2026 04:03:26 UTC (434 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.