Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
Quick Answer
Mage-VL is a codec-native streaming multimodal foundation model that improves real-time perception by over 75% in visual token consumption, outperforming existing models like Qwen3-VL-4B and Phi-4-reasoning-vision.
Quick Take
It achieves a 3.5x speedup in inference while addressing Moravec's paradox through a bio-inspired architecture and efficient data pipelines.
Key Points
- Mage-VL uses a custom tokenizer, Mage-ViT, to enhance efficiency.
- Trained on 560M images and 100M video frames, it matches top models.
- Achieves a 3.5x speedup in inference for video understanding tasks.
- Introduces AI4AI data pipelines for multimodal captioning optimization.
- Surpasses the 15B Phi-4-reasoning-vision baseline in performance.
DeepSignal Analysis
What happened
Mage-VL is a new multimodal foundation model designed for efficient real-time perception, achieving significant improvements in visual token consumption and inference speed. It employs a custom tokenizer and a dual-system architecture to enhance performance on streaming tasks.
Key evidence
- Mage-VL reduces visual token consumption by over 75% by using a custom tokenizer that selectively encodes dynamic regions in video frames.
- The model achieves a 3.5x speedup in inference compared to existing models, such as Qwen3-VL-4B and Phi-4-reasoning-vision.
- Mage-VL was trained on approximately 560 million unlabeled images and 100 million unlabeled video frames, matching or outperforming models trained on billions of image-text pairs.
Why it matters
The advancements presented by Mage-VL address the limitations of traditional vision-language models, particularly in real-time applications. By improving efficiency and performance in streaming perception tasks, it could enhance various applications, including video understanding and spatial reasoning, which are critical in fields like autonomous systems and interactive AI.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAuthors:Senqiao Yang, Kaichen Zhang, Zhaoyang Jia, Jinghao Guo, Yifei Shen, Xinjie Zhang, Xiaoyi Zhang, Haoqing Wang, Xiao Li, Peng Zhang, Xiang An, Yin Xie, Zhening Liu, Xun Guo, Jiahao Li, Shicheng Zheng, Jinglu Wang, Zongyu Guo, Wenxuan Xie, Zihan Zheng, Yuxuan Luo, Bin Li, Yan Lu
Abstract:Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.
| Comments: | Project page: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.24904 [cs.CV] |
| (or arXiv:2607.24904v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.24904 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Senqiao Yang [view email]
[v1]
Mon, 27 Jul 2026 17:59:53 UTC (2,571 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.