Residual Drift Dominates Contradiction in Multi-Turn Constraint Reasoning
Quick Take
The study reveals that multi-turn reasoning systems primarily fail due to satisfiable drift rather than logical contradictions. The DRIFT-Bench benchmark evaluates four models, with MUS-Repair outperforming others by 1.8 to 15.0 percentage points, highlighting that models often forget prior commitments instead of contradicting themselves. Reliable systems must ensure returned answers align with maintained states.
Key Points
- DRIFT-Bench includes 816 test problems across three constraint domains.
- MUS-Repair consistently outperforms the best non-MUS baseline by 1.8 to 15.0 pp.
- Residual errors in models show 98-100% satisfiable drift, with contradictions near zero.
- Models rarely contradict themselves but often forget prior commitments.
- Reliable multi-turn systems need to validate answers against maintained states.
Article Excerpt
From source RSS / original summaryarXiv:2605. 23940v1 Announce Type: new Abstract: How do multi-turn reasoning systems fail? The expected answer is logical contradiction, in which the system's maintained state becomes unsatisfiable. We show that the dominant mode is instead satisfiable drift, where the internal state stays consistent while the returned answer silently violates prior commitments.
We build DRIFT-Bench (Decomposing Reasoning Into Failure Types), a solver-instrumented benchmark of 816 test problems across three constraint domains, and evaluate four methods on it across four open-weight models (8B-120B parameters). MUS-Repair, which feeds minimal unsatisfiable subsets back to the generator, is strongest in every setting (+1. 8 to +15. 0 pp over the best non-MUS baseline). But the central finding is what repair leaves behind. After structured feedback, models rarely contradict themselves.
They forget. Residual errors are 98-100% satisfiable drift across all settings, while contradiction drops to near zero. Reliable multi-turn systems must separately validate that the returned answer respects the maintained state. Code is available at https://github. com/kaons-research/drift-bench.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
The Redpanda Agentic Data Plane (ADP) introduces out-of-band metadata channels to enhance the safety of autonomous AI agents, ensuring secure data access and tamper-proof audit trails. This architecture mitigates risks associated with unpredictable AI behavior by enforcing governance throughout the agent lifecycle, demonstrated in a multi-agent trading system with strict data scoping and approval thresholds.