Teach-and-Repeat: Accurately Extracting Operational Knowledge from Mobile Screen Demonstrations to Empower GUI Agents
Quick Answer
Teach VLM introduces a novel approach to extract operational knowledge from mobile screen demonstrations, significantly outperforming existing vision-language models in operation semantics prediction.
Quick Take
The Teach-and-Repeat paradigm enhances task automation for GUI agents, achieving improved Task Success Rates in Android environments.
Key Points
- Teach translates mobile screen trajectories into operational knowledge using keyframes from demonstration videos.
- A systematic data flywheel was developed to address the lack of aligned training data.
- The new Chinese Mobile Screen Teach Benchmark allows for fine-grained evaluation of the model's performance.
- Extensive evaluations show Teach VLM achieves state-of-the-art performance in operation semantics prediction.
- The Teach-and-Repeat paradigm improves Task Success Rates for downstream screen-based execution agents.
Paper Resources
Source Excerpt
arXiv:2606. 12817v1 Announce Type: new Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension. This capability enables models to convert visual state transitions into operational knowledge, defined as short natural-language sentences that describe action types, target UI elements, textual arguments, and execution orders.
However, due to the highly diverse and heterogeneous UI designs across applications, existing (VLMs) struggle to accurately infer these underlying operations. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.