Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement
Quick Answer
The MoTiF framework enhances interleaved thinking in multimodal models by addressing Modal Isolation through a two-stage training approach, significantly improving cross-modal coherence and task accuracy across four visual puzzle benchmarks.
Quick Take
The MoTiF framework enhances interleaved thinking in multimodal models by addressing Modal Isolation through a two-stage training approach, significantly improving cross-modal coherence and task accuracy across four visual puzzle benchmarks.
Key Points
- Identifies Modal Isolation as a key failure in interleaved thinking.
- Introduces modality transition loss to quantify cross-modal hallucination.
- MoTiF consists of Reflective SFT and Flow-GRPO training stages.
- Demonstrates substantial improvements in task accuracy across benchmarks.
- Emphasizes the need for structural supervision at modality boundaries.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 12886v1 Announce Type: new Abstract: Interleaved thinking, where a unified alternates between textual reasoning and visual generation, has shown promise on spatial and physical tasks. However, in complex long-chain scenarios, we identify a fundamental failure mode: generated images diverge from the textual context while subsequent text ignores the visual evidence, causing the two modalities to alternate without genuinely informing each other.
We term this Modal Isolation and attribute it to compounding information loss at modality boundaries. We decompose each reasoning cycle into atomic operations and define modality transition loss, quantifying cross-modal hallucination (text-to-image) and visual utilization deficit (image-to-text) at each boundary.
We propose MoTiF (Modality Tiransition Fidelity), a two-stage training framework that directly optimizes these transitions: Reflective SFT trains the model to detect and recover from erroneous visual outputs; Flow-GRPO improves image generation fidelity via reinforcement learning. All training signals in MoTiF derive from transition-level fidelity rather than end-task accuracy.
Across four visual puzzle benchmarks, this transition-level supervision substantially improves both cross-modal coherence and final task accuracy. The results demonstrate that effective interleaved reasoning requires explicit structural supervision at modality boundaries, not merely scaling or end-task optimization.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.