Video2LoRA: Parametric Video Internalization for Vision-Language Models
Quick Take
Video2LoRA introduces a novel method for parametric video internalization in vision-language models, enabling SmolVLM2 to answer queries with zero visual tokens. It reduces visual-token load by up to 1,500x and query TTFT by 6-80x while maintaining performance across multiple benchmarks.
Key Points
- Video2LoRA generates LoRA adapters directly from video in a single forward pass.
- Achieves equivalent performance to direct video-in-context inference across five captioning benchmarks.
- Reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x.
- Stable performance up to 1,024 frames and 1024px resolution.
- Supports independent adapter generation for non-overlapping video segments.
Article Content
From source RSS / original summaryarXiv:2606. 04351v1 Announce Type: new Abstract: Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Video2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass.
Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Video2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2. 2B on video summarization and captioning, Video2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time.
Video2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates.
Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Optimal Transport Flow Matching by Design
The study presents a novel approach to optimal transport (OT) flow matching, reformulating the problem by treating the prior as a design choice. This method achieves over 2x reduction in trajectory curvature compared to existing methods, improving generation quality in few-step regimes without altering the flow model. The approach integrates seamlessly with latent-space models and classifier-free guidance.