Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA
Quick Answer
This paper shows that A new RL framework trains LLMs on digital twin representations for surgical video QA, enhancing multi-step reasoning.
Quick Take
The approach achieves state-of-the-art results on the REAL-Colon-Reason benchmark with 2000 Q&A pairs, surpassing existing benchmarks like REAL-Colon-VQA and EndoVis18-VQA.
Key Points
- Introduces RL framework to decouple perception from reasoning in surgical video QA.
- Utilizes hierarchical representations with probabilistic uncertainty estimates.
- Achieves state-of-the-art performance on REAL-Colon-Reason benchmark.
- Includes 2000 question-answer pairs across three complexity levels.
- Novel reward combines format validation and clinical plausibility evaluation.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 17279v1 Announce Type: new Abstract: Surgical video question answering requires multi-step reasoning across semantic, spatial, and temporal dimensions. Existing methods architecturally compress videos into discrete token representations and couple visual perception with reasoning. This approach fragments continuous spatial-temporal relationships and has been shown to restrict multi-step reasoning capabilities.
We introduce a reinforcement learning (RL) framework that trains (LLMs) to decouple perception from reasoning by operating over digital twin representations constructed from surgical foundation models. Additionally, we introduce hierarchical representations across frame, temporal window, and procedure levels with probabilistic uncertainty estimates.
Finally, we propose a novel reward that combines format validation with accuracy assessment through clinical plausibility evaluation and uncertainty-aware calibration for training. To demonstrate the capabilities of this approach, we introduce REAL-Colon-Reason, a colonoscopic benchmark with 2000 question-answer pairs across three complexity levels. We achieve state-of-the-art performance on REAL-Colon-Reason and two existing surgical VideoQA benchmarks REAL-Colon-VQA and EndoVis18-VQA.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.