Constructing Evaluation Datasets for Procedural Reasoning: Balancing Naturalness, Grounding, and Multi-Hop Coverage
Quick Answer
The study evaluates procedural reasoning datasets generated using three TMK-based strategies, revealing that strict TMK generation yields the highest quality with 96.5% grounded questions.
Quick Take
The study evaluates procedural reasoning datasets generated using three TMK-based strategies, revealing that strict TMK generation yields the highest quality with 96.5% grounded questions. Transcript-first generation offers more learner-like questions but suffers from weak grounding, while TMK-aware generation excels in multi-hop coverage but lacks grounding. These findings highlight the need for representation-aware validation in AI-supported learning systems.
Key Points
- Strict TMK generation achieves 96.5% grounded questions and 92.6% usable questions.
- Transcript-first generation produces more learner-like but context-dependent questions.
- TMK-aware generation has high multi-hop coverage but lower grounding quality.
- The study covers 23 instructional topics and 690 question-answer pairs.
- Findings emphasize the importance of representation-aware validation in dataset evaluation.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 12767v1 Announce Type: new Abstract: Evaluating procedural reasoning in AI-supported learning systems requires question-answer datasets that are both learner-like and grounded in the instructional knowledge the system is expected to use. We study how TMK-based question generation strategies affect dataset quality for procedural and multi-hop reasoning.
We compare three strategies: strict generation from Task-Method-Knowledge (TMK) models, transcript-first generation with post-hoc TMK filtering, and TMK-aware generation that combines transcripts with structured guidance. To evaluate generated items, we introduce a grounding validation framework based on closed-set evidence units extracted from TMK models.
The framework measures whether answers are supported by the underlying representation, whether questions are self-contained, and whether they target multi-hop procedural reasoning. Across 23 instructional topics and 690 generated question-answer pairs, strict TMK generation achieves the strongest overall quality, with 96. 5% grounded questions and 92. 6% usable questions.
Transcript-first generation produces more learner-like questions but more context-dependent or weakly grounded items, while TMK-aware generation yields high raw multi-hop coverage but lower grounding. These results show that procedural richness and natural phrasing do not guarantee representational grounding, motivating explicit representation-aware validation for evaluation datasets in AI-supported learning.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Arbor: Tree Search as a Cognition Layer for Autonomous Agents
Arbor introduces a multi-agent framework utilizing structured tree search for optimizing LLM inference, achieving up to 193% throughput-latency improvement compared to vendor-optimized systems. It employs an Orchestrator and Critic agent for stability and coordination, demonstrating hardware-agnostic performance with minimal variance.