Benchmarking Agentic Review Systems
Quick Answer
A study evaluates agentic review systems, finding OpenAIReview + GPT-5.5 achieves 83.0% accuracy in assessing paper quality and detects 71.6% of injected errors.
Quick Take
Real user feedback indicates positive reception but highlights issues with false positives.
Key Points
- OpenAIReview + GPT-5.5 outperforms other systems with 83.0% accuracy in peer review.
- The system detects 71.6% of errors in papers with injected perturbations.
- Combined detection across six models reaches 83.3% recall, indicating varied error detection capabilities.
- User feedback shows a positive vote ratio of 1.44 to 1, but highlights false positives.
- The study suggests improvements are needed for AI review systems despite their current effectiveness.
Paper Resources
Source Excerpt
A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated. We evaluate two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline, across six spanning frontier and efficient models. First, we study whether AI reviews on ICLR/NeurIPS papers track with papers' quality as approximated by external signals such as citation
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.