Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA
Quick Answer
This paper shows that A new RL framework trains LLMs on digital twin representations for surgical video QA, enhancing multi-step reasoning.
Quick Take
A new RL framework trains LLMs on digital twin representations for surgical video QA, enhancing multi-step reasoning. The approach achieves state-of-the-art results on the REAL-Colon-Reason benchmark with 2000 Q&A pairs, surpassing existing benchmarks like REAL-Colon-VQA and EndoVis18-VQA.
Key Points
- Introduces RL framework to decouple perception from reasoning in surgical video QA.
- Utilizes hierarchical representations with probabilistic uncertainty estimates.
- Achieves state-of-the-art performance on REAL-Colon-Reason benchmark.
- Includes 2000 question-answer pairs across three complexity levels.
- Novel reward combines format validation and clinical plausibility evaluation.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 17279v1 Announce Type: new Abstract: Surgical video question answering requires multi-step reasoning across semantic, spatial, and temporal dimensions. Existing methods architecturally compress videos into discrete token representations and couple visual perception with reasoning. This approach fragments continuous spatial-temporal relationships and has been shown to restrict multi-step reasoning capabilities.
We introduce a reinforcement learning (RL) framework that trains large language models (LLMs) to decouple perception from reasoning by operating over digital twin representations constructed from surgical foundation models. Additionally, we introduce hierarchical representations across frame, temporal window, and procedure levels with probabilistic uncertainty estimates.
Finally, we propose a novel reward that combines format validation with accuracy assessment through clinical plausibility evaluation and uncertainty-aware calibration for training. To demonstrate the capabilities of this approach, we introduce REAL-Colon-Reason, a colonoscopic benchmark with 2000 question-answer pairs across three complexity levels. We achieve state-of-the-art performance on REAL-Colon-Reason and two existing surgical VideoQA benchmarks REAL-Colon-VQA and EndoVis18-VQA.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.