RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review
Quick Answer
RubricReviewer introduces a rubric-driven framework for peer review, combining a training-free agent (Scout) and a human-aligned model (Aligner) to enhance review comprehensiveness and robustness against adversarial attacks.
Quick Take
Experiments show it outperforms prior systems in generating more discriminative reviews.
Key Points
- RubricReviewer explicitly generates adaptive rubrics for peer review.
- Combines training-free evidence gathering with human-aligned review generation.
- Produces more comprehensive and discriminative reviews than previous systems.
- Demonstrates strong robustness against adversarial prompt-injection attacks.
- Ablation studies confirm the necessity of each component in the framework.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants. Existing LLM-based reviewers, however, face two structural limitations. First, they map manuscripts directly to reviews, leaving the underlying rubric implicit and entangling its derivation with the judgement. Second, the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but produce undirected critiques, while training-based reviewers inherit human discriminative judgement together with its noise and uneven coverage. We introduce RubricReviewer, a fully rubric-driven framework that addresses both limitations. It makes rubric generation an explicit intermediate step, so that both review generation and the final assessment are conditioned on paper-adaptive rubrics. It further combines a training-free agent (Scout) that gathers external evidence with a human-aligned trained model (Aligner) that consumes this evidence, fusing the strengths of both supervision sources. Experiments on real-world submissions show that RubricReviewer produces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt-injection attacks. Ablation studies further confirm the necessity of each component.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.00005 [cs.CL] |
| (or arXiv:2608.00005v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.00005 arXiv-issued DOI via DataCite |
Submission history
From: Shuyu Guo [view email]
[v1]
Thu, 4 Jun 2026 10:47:08 UTC (215 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.