CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Quick Answer
CausalDS introduces a novel benchmark for evaluating causal reasoning in data-science agents, integrating structural causal models with observational data and synthetic narratives.
Quick Take
This benchmark spans Pearl's causal reasoning rungs and includes tasks that require data science coding, uncertainty quantification, and , addressing the limitations of existing datasets.
Key Points
- CausalDS benchmarks causal reasoning in data-science workflows using structural causal models.
- Tasks include data science coding, uncertainty quantification, and recognizing unanswerable questions.
- Synthetic narratives are grounded in realistic domains, enhancing the benchmark's applicability.
- The benchmark aims to reduce risks associated with 'causal parroting' in AI models.
- It addresses gaps in existing benchmarks by combining causal reasoning with realistic data analysis.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. We optionally ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the "causal parrot" risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs, with typical data-science prediction tasks appearing as Rung 1. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first-class scored outcome. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.
| Comments: | 55 pages, 10 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.08093 [cs.AI] |
| (or arXiv:2607.08093v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.08093 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Andrej Leban [view email]
[v1]
Thu, 9 Jul 2026 04:03:26 UTC (434 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.