Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA
Quick Answer
This paper shows that A new RL framework trains LLMs on digital twin representations for surgical video QA, enhancing multi-step reasoning.
Quick Take
The approach achieves state-of-the-art results on the REAL-Colon-Reason benchmark with 2000 Q&A pairs, surpassing existing benchmarks like REAL-Colon-VQA and EndoVis18-VQA.
Key Points
- Introduces RL framework to decouple perception from reasoning in surgical video QA.
- Utilizes hierarchical representations with probabilistic uncertainty estimates.
- Achieves state-of-the-art performance on REAL-Colon-Reason benchmark.
- Includes 2000 question-answer pairs across three complexity levels.
- Novel reward combines format validation and clinical plausibility evaluation.
Paper Resources
Source Excerpt
arXiv:2606. 17279v1 Announce Type: new Abstract: Surgical video question answering requires multi-step reasoning across semantic, spatial, and temporal dimensions. Existing methods architecturally compress videos into discrete token representations and couple visual perception with reasoning. This approach fragments continuous spatial-temporal relationships and has been shown to restrict multi-step reasoning capabilities.
We introduce a reinforcement learning (RL) framework that trains (LLMs) to decouple perception from reasoning by operating over digital twin representations constructed from surgical foundation models. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.