Syll: Open-Source Personal Automation with Cross-Surface Execution
Quick Answer
Syll is an open-source multimodal personal automation agent that integrates APIs, CLI, and GUI, enabling users to teach and audit agent behavior across diverse interfaces.
Quick Take
It supports direct user demonstrations to compile reusable skills and provides multimodal evidence for inspection, validated on applications like Adobe Photoshop and macOS Finder.
Key Points
- Syll unifies /API tools, CLI execution, and visual GUI control in a modular runtime.
- Users can teach procedures through demonstrations, which Syll converts into reusable skills.
- Agent execution generates multimodal evidence for user inspection and control.
- Memory, skills, and routines are externalized as editable local artifacts.
- Validated on production applications like Adobe Photoshop and Stardew Valley.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 07594v1 Announce Type: new Abstract: Personal AI agents must increasingly operate across APIs, shells, web surfaces, and desktop GUIs, yet many systems remain tuned to a single interface and offer limited support for user teaching and auditability.
We present Syll, an open-source, self-hosted multimodal agent harness that unifies /API tools, CLI execution, and visual GUI control in a modular runtime, enabling agents to coordinate computer use across heterogeneous interfaces while streamlining how users and agents exchange information.
At the core of Syll is a bidirectional user-agent interaction layer: users teach procedures through direct demonstration, which Syll compiles into reusable skills; agent execution is translated back into multimodal evidence -- logs, keyframes, and approval checkpoints -- for inspection and control. Syll further externalizes memory, skills, routines, and governance as editable local artifacts, supporting straightforward inspection, extension, and downstream development.
Our implementation has been validated on production desktop applications including Adobe Photoshop, Adobe Audition, Stardew Valley, macOS Finder and others. We report mechanism-oriented studies that validate multimodal routing, teachable GUI replay, and persistent local artifacts. We hope Syll can serve as a practical open-source foundation for personal automation that users can teach, inspect, and continuously extend.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.