DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding
Quick Take
DRS-GUI is a training-free framework that enhances GUI grounding using dynamic region search techniques.
Key Points
- Introduces a lightweight UI Perceptor for effective region proposals.
- Employs Monte Carlo Tree Search for dynamic action scheduling.
- Achieves 14% improvement in grounding performance on ScreenSpot-Pro.
📖 Reader Mode
~2 min readAbstract:GUI agents powered by Multimodal Large Language Models (MLLMs) have demonstrated impressive capability in understanding and executing user instructions. However, accurately grounding instruction-relevant elements from high-resolution screenshots cluttered with irrelevant UI components remains challenging for existing approaches. Inspired by how humans dynamically adjust their perceptual scope to locate task-related regions on complex screens, we propose DRS-GUI, a training-free dynamic region search framework for GUI grounding that can be seamlessly integrated into existing MLLMs. DRS-GUI introduces a lightweight UI Perceptor that performs three human-like perceptual actions (Focus, Shift, and Scatter) to progressively explore the interface and generate region proposals. To dynamically schedule these actions, we further design an Action Planner based on Monte Carlo Tree Search (MCTS). A region quality reward is employed to evaluate and select the highly instruction-relevant region, efficiently pruning redundant UI elements. Experiments demonstrate that DRS-GUI yields a 14\% improvement on ScreenSpot-Pro for general and GUI-specific MLLMs (Qwen2.5-VL-7B and UGround-V1-7B), significantly enhancing grounding performance and generalization.
| Comments: | 11 pages, 8 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.15542 [cs.AI] |
| (or arXiv:2605.15542v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15542 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yichao Liu [view email]
[v1]
Fri, 15 May 2026 02:27:41 UTC (7,996 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →From Prompts to Protocols: An AI Agent for Laboratory Automation
An AI agent integrates large language models for automating laboratory protocols, enhancing efficiency and accuracy.