Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix
Quick Answer
This study reveals that goal-conditioned predictors can achieve high accuracy through instruction transcription rather than genuine perception, leading to significant instruction leakage.
Quick Take
This study reveals that goal-conditioned predictors can achieve high accuracy through instruction transcription rather than genuine perception, leading to significant instruction leakage. By excluding goals from dynamics and supervising the read path, the model can recover true grounding, achieving an accuracy of 0.88, consistent regardless of goal presence.
Key Points
- Goal-conditioned predictors reached 0.90 accuracy but relied on instruction transcription.
- Withholding goals dropped accuracy to chance levels (0.90 to 0.27).
- Counterfactual instructions misled predictions 94.5% of the time.
- Excluding goals from dynamics improved grounding accuracy to 0.88.
- The findings apply to any goal-conditioned world model with similar instruction issues.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}. We ask when such references actually ground a relation, and identify a trap: a goal-conditioned predictor reaches a striking $0.90$ relation-readout accuracy, yet this is \emph{instruction transcription}, not perception. Withholding the goal collapses it to chance ($0.90\!\to\!0.27$, three seeds) and a counterfactual instruction makes the predicted anchors follow the \emph{false} instruction $94.5\%$ of the time (true scene $2.3\%$; $N{=}256$). Tested across three settings and a within-task ablation, our central claim characterizes the confound: \textbf{instruction leakage occurs when the scored quantity is transcribable from the instruction (when the instruction names the answer) and is essentially independent of how predictive the non-instruction inputs are.} Our tabletop and the external BabyAI benchmark leak, whereas a Language-Table forward-dynamics world model whose instruction names \emph{referents} does not, until the instruction is augmented to name the direction; and degrading the action never increases leakage, the opposite of what predictor-competition predicts. The diagnosis prescribes the fix: keep the goal out of the dynamics (it belongs to the planner's cost) and supervise the \emph{read} path, recovering genuine, instruction-independent grounding ($0.88$, identical with and without the goal). The detection protocol and remedy apply to any goal-conditioned world model whose instruction names the scored quantity.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.06925 [cs.AI] |
| (or arXiv:2607.06925v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.06925 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yufeng Wang [view email]
[v1]
Wed, 8 Jul 2026 02:38:43 UTC (76 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Onnes: A Physics-Grounded LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure
Onnes is a physics-grounded digital twin simulator for dilution refrigerators, enhancing cryogenic fault diagnosis in quantum computing. It achieves a classification accuracy of 99.0% using few-shot demonstrations, matching a supervised ML classifier, while maintaining a low false alarm rate of 6.4% on real hardware.