Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution
Quick Answer
The paper presents interventional grounding audits for evaluating premise dependency in LLMs, achieving an F1 score of 0.806 on ProntoQA with GPT-4o.
Quick Take
The paper presents interventional grounding audits for evaluating premise dependency in LLMs, achieving an F1 score of 0.806 on ProntoQA with GPT-4o. This method outperforms self-consistency baselines and reveals that 66% of solved problems show insensitivity to direct proof-tree dependencies, highlighting a significant blind spot in current evaluation methods. All results and scripts are publicly available.
Key Points
- Interventional grounding audits test premise dependency in LLMs using predicate substitution.
- Achieved F1 score of 0.806 on 50 ProntoQA problems with GPT-4o.
- Outperformed self-consistency baseline (F1 = 0.343) significantly.
- 66% of correctly solved problems showed insensitivity to direct proof-tree dependencies.
- Audit certificates and scripts are available in a public GitHub repository.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language models produce chain-of-thought (CoT) reasoning that appears logically sound yet may not genuinely depend on its stated premises. We introduce interventional grounding audits, a black-box, step-level test of premise dependency: we intervene on a single premise by substituting its target predicate with a fresh symbol, re-run the model, and check whether each reasoning step's normalized conclusion (canonical predicate form) changes. We evaluate on ProntoQA, a synthetic multi-hop deductive reasoning benchmark with gold proof trees, where step-level premise dependencies are known. Applied to 50 ProntoQA problems with GPT-4o, our method achieves F1 = 0.806 on detecting proof-tree dependencies (F1 = 0.885 on predicate-determining dependencies; Recall = 100%), significantly outperforming a self-consistency baseline (F1 = 0.343; 95% bootstrap CIs non-overlapping). We further identify that 66% of correctly-solved problems contain at least one aligned step insensitive to a direct proof-tree dependency under consistent substitution -- all involving entity-introduction premises, a documented blind spot of the consistent-substitution evaluator -- a "right answer, wrong reasoning" signal invisible to passive methods. All audit certificates, raw outputs, and reproduction scripts are available in a public GitHub repository, and we discuss scope limits beyond formal, parsable benchmarks.
| Comments: | Accepted at the ICLR 2026 Workshop on Logical Reasoning of Large Language Models (this https URL) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Logic in Computer Science (cs.LO) |
| Cite as: | arXiv:2607.13069 [cs.AI] |
| (or arXiv:2607.13069v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13069 arXiv-issued DOI via DataCite |
Submission history
From: Hironao Nakamura [view email]
[v1]
Sat, 11 Jul 2026 01:32:22 UTC (42 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System
The MEDA system utilizes large language models and symbolic regression to autonomously discover ordinary differential equations for biological systems, achieving strong structural recovery and biologically plausible models. It outperforms existing methods by integrating domain knowledge and mechanistic constraints, demonstrating effective retrieval and extrapolation capabilities.