Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
Quick Answer
Mage-VL is a codec-native streaming multimodal foundation model that improves real-time perception by over 75% in visual token consumption, outperforming existing models like Qwen3-VL-4B and Phi-4-reasoning-vision.
Quick Take
It achieves a 3.5x speedup in inference while addressing Moravec's paradox through a bio-inspired architecture and efficient data pipelines.
Key Points
- Mage-VL uses a custom tokenizer, Mage-ViT, to enhance efficiency.
- Trained on 560M images and 100M video frames, it matches top models.
- Achieves a 3.5x speedup in inference for video understanding tasks.
- Introduces AI4AI data pipelines for multimodal captioning optimization.
- Surpasses the 15B Phi-4-reasoning-vision baseline in performance.
DeepSignal Analysis
What happened
Mage-VL is a new multimodal foundation model designed for efficient real-time perception, achieving significant improvements in visual token consumption and inference speed. It employs a custom tokenizer and a dual-system architecture to enhance performance on streaming tasks.
Key evidence
- Mage-VL reduces visual token consumption by over 75% by using a custom tokenizer that selectively encodes dynamic regions in video frames.
- The model achieves a 3.5x speedup in inference compared to existing models, such as Qwen3-VL-4B and Phi-4-reasoning-vision.
- Mage-VL was trained on approximately 560 million unlabeled images and 100 million unlabeled video frames, matching or outperforming models trained on billions of image-text pairs.
Why it matters
The advancements presented by Mage-VL address the limitations of traditional vision-language models, particularly in real-time applications. By improving efficiency and performance in streaming perception tasks, it could enhance various applications, including video understanding and spatial reasoning, which are critical in fields like autonomous systems and interactive AI.
What to watch
Paper Resources
Source Excerpt
Standard (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energ
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.