iFLYTEK-Embodied-Omni Technical Report
Quick Answer
This paper shows that iFLYTEK-Embodied-Omni is a unified multimodal foundation model that integrates vision, language, and action for general-purpose embodied agents, addressing limitations of existing approaches.
Quick Take
iFLYTEK-Embodied-Omni is a unified multimodal foundation model that integrates vision, language, and action for general-purpose embodied agents, addressing limitations of existing approaches. It employs a four-stage training strategy using a comprehensive dataset of action-annotated and action-free embodied videos, enhancing instruction understanding and action execution.
Key Points
- The model features a brain-cerebellum collaboration for improved task planning and execution.
- It combines visual-language, video-generation, and action-generation components within a single framework.
- Training involves action-annotated and action-free videos from human and robot interactions.
- The four-stage strategy progressively trains individual components before joint fine-tuning.
- This approach aims to reduce prediction errors in multimodal instruction processing.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Yuan Zhang, Jingfei Ni, Guanchen Lu, Shiqi Zhang, Qingshan Xu, Chi Liu, Xin Nie, Wenjie Xu, Lin Gao, Zhiyuan Cheng, Mingxin Zhou, Jiajia Wu, Diyuan Liu, Jia Pan, Chao Ji
Abstract:General-purpose embodied agents must understand multimodal instructions, anticipate how their environment will evolve, and produce precise control actions over extended horizons. Existing approaches typically specialize in visual-language reasoning, video-based world modeling, or action generation, while cascaded pipelines that first synthesize future observations and then infer actions can introduce interface bottlenecks and compound prediction errors. We present iFLYTEK-Embodied-Omni, a unified multimodal foundation model that jointly models vision(videos and images), language, and action within a single Omni framework. Its modality-specific visual-language, video-generation, and action-generation components communicate through shared multimodal self-attention. This design establishes brain-cerebellum collaboration: the vision-language modeland video generation model form a high-level brain for instruction understanding, task planning, progress tracking, and future visual-state prediction, whereas the action generation modelserves as a low-level cerebellum that directly converts planned subgoals and shared multimodal context into executable action chunks. To develop these capabilities, we combine action-annotated and action-free embodied videos from human demonstrations and robot interactions with embodied reasoning, embodied perception, and general-purpose image-text data to construct a comprehensive dataset. We further adopt a four-stage strategy that progressively trains the VLM, VGM, and AGM before jointly fine-tuning the complete model.
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.02542 [cs.AI] |
| (or arXiv:2607.02542v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02542 arXiv-issued DOI via DataCite |
Submission history
From: Yuan Zhang [view email]
[v1]
Wed, 24 Jun 2026 00:25:44 UTC (3,114 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Procedural Memory Distillation (PMD) enhances reinforcement learning by converting cross-episode signals into reusable memory, improving Qwen3-8B and OLMo3-Instruct-7B models by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on . The co-evolution of policy and memory allows for more effective self-supervision, demonstrating significant performance gains when both components are active.