PRO-CUA: Process-Reward Optimization for Computer Use Agents
Quick Take
PRO-CUA introduces a novel process-reward optimization framework for training computer use agents (CUAs), enhancing their learning efficiency by decoupling environment interaction from policy optimization. This method utilizes step-level feedback from a process reward model (PRM) to mitigate imitation bottlenecks and improve credit assignment, demonstrating effectiveness in live web benchmarks.
Key Points
- PRO-CUA enhances CUA training by using iterative step-level reinforcement learning.
- Decouples on-policy interaction from policy optimization for improved efficiency.
- Utilizes process reward models to provide dense feedback and reduce distribution shift.
- Demonstrated effectiveness through experiments on live web benchmarks.
- Addresses challenges like sparse rewards and ambiguous credit assignment.
Article Content
From source RSS / original summaryarXiv:2605. 29119v1 Announce Type: new Abstract: Computer use agents (CUAs) have shown strong potential for automating complex digital workflows, yet their training remains constrained by costly live environment interaction and limited high-quality supervision. Existing filtered behavior cloning pipelines suffer from imitation bottlenecks, including distribution shift from the expert demonstration and the absence of negative learning signals.
Meanwhile, standard trajectory-level reinforcement learning struggles with sparse rewards, ambiguous credit assignment, and high infrastructure costs for long-horizon GUI interaction. In this work, we propose PRO-CUA, a process-reward optimization framework for training CUAs with iterative step-level reinforcement learning.
PRO-CUA decouples on-policy environment interaction from policy optimization: the current policy collects states through live rollouts, generates diverse candidate actions for each state, receives step-level feedback from a process reward model (PRM), and is optimized with group-relative advantages. This design enables dense and flexible credit assignment without relying on golden answers or offline expert trajectories, while reducing distribution shift by training on the agent's own execution states.
Experiments on live web benchmarks demonstrate the effectiveness of PRO-CUA and the reliability of PRM-guided step-level training.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
The Redpanda Agentic Data Plane (ADP) introduces out-of-band metadata channels to enhance the safety of autonomous AI agents, ensuring secure data access and tamper-proof audit trails. This architecture mitigates risks associated with unpredictable AI behavior by enforcing governance throughout the agent lifecycle, demonstrated in a multi-agent trading system with strict data scoping and approval thresholds.