Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
Quick Take
The proposed Guidance Contrastive Policy Optimization (GCPO) algorithm enhances discrete policy optimization by enabling per-token credit assignment, outperforming GRPO and DAPO in text-to-image generation and reasoning tasks. GCPO provides more precise learning signals by contrasting model predictions, emphasizing semantically relevant areas, thus improving performance in diverse applications.
Key Points
- GCPO enables per-token credit assignment, addressing limitations of sample-level rewards.
- The algorithm contrasts predictions under positive and negative prompts for better learning signals.
- Empirical results show GCPO outperforms GRPO and DAPO on key benchmarks.
- GCPO emphasizes semantically relevant regions in text-to-image generation tasks.
- The method is scalable and effective for discrete policy learning across various domains.
Article Content
From source RSS / original summaryarXiv:2605. 29198v1 Announce Type: new Abstract: Group-advantage-based reinforcement learning methods, such as GRPO and DAPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation as uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions.
To address this issue, we propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals.
Empirically, we find that GCPO emphasizes semantically relevant regions such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, achieving 0.11% parameter updates while enhancing uncertainty-aware fine-tuning. It outperforms state-of-the-art methods across 15 biomedical imaging datasets, proving effective in few-shot learning and domain shifts for clinical applications.