VISUALSKILL: Multimodal Skills for Computer-Use Agents
Quick Answer
VISUALSKILL enhances computer-use agents (CUAs) by integrating visual elements into skill artifacts, achieving a 15.3-point improvement on CUA benchmarks.
Quick Take
A Claude Code CLI agent using VISUALSKILL scored 0.456, outperforming text-only skills by 8.3 points, demonstrating the importance of visual context in UI interactions.
Key Points
- VISUALSKILL organizes multimodal skills tailored to specific applications.
- Claude Code CLI agent achieved an average score of 0.456 on CUA benchmarks.
- The approach yielded a 15.3-point lift over the no-skill baseline score of 0.303.
- VISUALSKILL outperformed a matched text-only skill by 8.3 points.
- The integration of visual figures aids in UI element identification and workflow verification.
Paper Resources
Source Excerpt
Computer-use agents (CUAs) approach human-level performance on standardised benchmarks but still struggle on long-horizon tasks and unseen software. Existing skill libraries address this with reusable skills, but represent the skill artifact as text only, despite the visual nature of GUI interaction. We propose VISUALSKILL: a hierarchical multimodal skill, tailored to each target application and organised as a central index over per-topic files, which the agent consumes through a load_topic
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.