VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
Quick Answer
VideoSEG-O3 introduces a multi-turn reinforcement learning framework for reasoning video object segmentation, enhancing pixel-level localization through a coarse-to-fine cognitive process.
Quick Take
VideoSEG-O3 introduces a multi-turn reinforcement learning framework for reasoning video object segmentation, enhancing pixel-level localization through a coarse-to-fine cognitive process. It integrates SEG-aware logit calibration for improved segmentation quality and features a specialized dataset, VTS-CoT, to support comprehensive reasoning trajectories.
Key Points
- VideoSEG-O3 is the first framework to use multi-turn reinforcement learning for RVOS.
- It employs a multi-turn temporal-spatial chain-of-thought for fine-grained detail capture.
- SEG-aware logit calibration integrates pixel-wise feedback into token-level logits.
- The VTS-CoT dataset features comprehensive reasoning trajectories for enhanced training.
- Code and models will be available on GitHub for public access.
Article Content
From source RSS / original summaryarXiv:2606. 06819v1 Announce Type: new Abstract: Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos.
To address this, we propose \textbf{VideoSEG-O3}, the first multi-turn reinforcement learning framework for RVOS that emulates the human \textit{``coarse-to-fine''} cognitive process. It employs a \textit{multi-turn temporal-spatial chain-of-thought} to capture fine-grained details by iteratively pinpointing critical intervals and keyframes.
Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of \texttt{[SEG]} during the RL stage, we introduce \textit{SEG-aware logit calibration}, which integrates pixel-wise segmentation feedback directly into the token-level logits.
Furthermore, we design a \textit{decoupled thinking trace} to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct \textbf{VTS-CoT}, a specialized cold-start dataset featuring comprehensive reasoning trajectories. The code and models will be released at https://github. com/Dmmm1997/VideoSEG-O3.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.
