Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution
Quick Answer
The paper presents interventional grounding audits for evaluating premise dependency in LLMs, achieving an F1 score of 0.806 on ProntoQA with GPT-4o.
Quick Take
This method outperforms self-consistency baselines and reveals that 66% of solved problems show insensitivity to direct proof-tree dependencies, highlighting a significant blind spot in current evaluation methods. All results and scripts are publicly available.
Key Points
- Interventional grounding audits test premise dependency in using predicate substitution.
- Achieved F1 score of 0.806 on 50 ProntoQA problems with GPT-4o.
- Outperformed self-consistency baseline (F1 = 0.343) significantly.
- 66% of correctly solved problems showed insensitivity to direct proof-tree dependencies.
- Audit certificates and scripts are available in a public GitHub repository.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language models produce chain-of-thought (CoT) reasoning that appears logically sound yet may not genuinely depend on its stated premises. We introduce interventional grounding audits, a black-box, step-level test of premise dependency: we intervene on a single premise by substituting its target predicate with a fresh symbol, re-run the model, and check whether each reasoning step's normalized conclusion (canonical predicate form) changes. We evaluate on ProntoQA, a synthetic multi-hop deductive reasoning benchmark with gold proof trees, where step-level premise dependencies are known. Applied to 50 ProntoQA problems with GPT-4o, our method achieves F1 = 0.806 on detecting proof-tree dependencies (F1 = 0.885 on predicate-determining dependencies; Recall = 100%), significantly outperforming a self-consistency baseline (F1 = 0.343; 95% bootstrap CIs non-overlapping). We further identify that 66% of correctly-solved problems contain at least one aligned step insensitive to a direct proof-tree dependency under consistent substitution -- all involving entity-introduction premises, a documented blind spot of the consistent-substitution evaluator -- a "right answer, wrong reasoning" signal invisible to passive methods. All audit certificates, raw outputs, and reproduction scripts are available in a public GitHub repository, and we discuss scope limits beyond formal, parsable benchmarks.
| Comments: | Accepted at the ICLR 2026 Workshop on Logical Reasoning of Large Language Models (this https URL) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Logic in Computer Science (cs.LO) |
| Cite as: | arXiv:2607.13069 [cs.AI] |
| (or arXiv:2607.13069v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13069 arXiv-issued DOI via DataCite |
Submission history
From: Hironao Nakamura [view email]
[v1]
Sat, 11 Jul 2026 01:32:22 UTC (42 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.