DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding
Quick Answer
DRS-GUI is a training-free framework that enhances GUI grounding for MLLMs, achieving a 14% improvement on the ScreenSpot-Pro benchmark with models like Qwen2.5-VL-7B and UGround-V1-7B.
Quick Take
It utilizes a lightweight UI Perceptor and an Action Planner based on Monte Carlo Tree Search to efficiently identify relevant UI regions, significantly improving grounding performance and generalization.
Key Points
- Introduces a lightweight UI Perceptor for dynamic region search in GUI grounding.
- Utilizes Monte Carlo Tree Search for scheduling perceptual actions effectively.
- Achieves a 14% performance improvement on ScreenSpot-Pro benchmark.
- Enhances grounding performance for general and GUI-specific MLLMs.
- Prunes redundant UI elements to focus on instruction-relevant regions.
Paper Resources
📖 Reader Mode
~2 min readAbstract:GUI agents powered by Multimodal Large Language Models (MLLMs) have demonstrated impressive capability in understanding and executing user instructions. However, accurately grounding instruction-relevant elements from high-resolution screenshots cluttered with irrelevant UI components remains challenging for existing approaches. Inspired by how humans dynamically adjust their perceptual scope to locate task-related regions on complex screens, we propose DRS-GUI, a training-free dynamic region search framework for GUI grounding that can be seamlessly integrated into existing MLLMs. DRS-GUI introduces a lightweight UI Perceptor that performs three human-like perceptual actions (Focus, Shift, and Scatter) to progressively explore the interface and generate region proposals. To dynamically schedule these actions, we further design an Action Planner based on Monte Carlo Tree Search (MCTS). A region quality reward is employed to evaluate and select the highly instruction-relevant region, efficiently pruning redundant UI elements. Experiments demonstrate that DRS-GUI yields a 14\% improvement on ScreenSpot-Pro for general and GUI-specific MLLMs (Qwen2.5-VL-7B and UGround-V1-7B), significantly enhancing grounding performance and generalization.
| Comments: | 11 pages, 8 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.15542 [cs.AI] |
| (or arXiv:2605.15542v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15542 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yichao Liu [view email]
[v1]
Fri, 15 May 2026 02:27:41 UTC (7,996 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.