Reflective Dialogue or Prompt Refinement? Effects of Tutor Scaffolding on Students' Independent LLM Use for Programming
Quick Answer
This study compares two LLM-based tutoring approaches—Socratic-Guidance (SG) and Prompt-Refinement (PR)—in a graduate robotics course.
Quick Take
While both methods yielded similar task performance, SG students demonstrated greater learning gains and better prompting strategies in subsequent use, emphasizing the importance of Socratic guidance in educational LLM design.
Key Points
- 66 graduate students participated in a 6-week intervention using SG or PR tutors.
- SG students achieved higher learning gains compared to PR students in later sessions.
- Socratic guidance led to understanding-driven prompting strategies in unconstrained LLM use.
- Students perceived SG as less efficient but it fostered better long-term learning.
- Findings highlight the need for effective LLM tutor design incorporating Socratic methods.
Paper Resources
📖 Reader Mode
~2 min readAbstract:While Large Language Models (LLMs) can provide personalized support in learning, several studies have raised concerns regarding their use in education. Importantly, learning depends on how students engage with LLMs. This study examined how two types of LLM-based tutors shape students' prompting practices, learning, and subsequent LLM-use: a Socratic-Guidance (SG) tutor, which structures interaction through dialogic questioning, and a Prompt-Refinement (PR) tutor that guides the formulation of effective prompts. We conducted a two-phase study in a graduate-level mobile robotics course: 66 students used either the SG or PR tutor during a 6-week intervention, followed by 52 students using an unconstrained LLM during a 3-week course project. Results show that while the SG- and PR tutors led to similar task performance and prompting patterns during guided use, they differ in learning outcomes and later LLM-use. SG-students, relative to PR-student, achieved higher learning gains in later sessions, and were more likely to adopt understanding-driven prompting strategies, which are predictive of higher understanding, when using an unconstrained LLM. Although learners perceived the SG tutor as less efficient, the findings suggest that Socratic guidance supports the development of students' capacity to learn with LLMs over time, highlighting its importance for LLM tutor design.
| Comments: | Best paper award, Published in AIED 2026: The 27th International Conference on Artificial Intelligence in Education |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.03303 [cs.AI] |
| (or arXiv:2607.03303v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.03303 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | In International Conference on Artificial Intelligence in Education (pp. 546-561). Cham: Springer Nature Switzerland (2026) |
| Related DOI: | https://doi.org/10.1007/978-3-032-29763-1_37
DOI(s) linking to related resources |
Submission history
From: Jérôme Brender [view email]
[v1]
Fri, 3 Jul 2026 13:15:36 UTC (1,653 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.