Debate Helps Weak Judges Reward Stronger Models
Quick Answer
This paper shows that In proposer-critic debate settings, stronger critics can significantly enhance judge performance on code and logic tasks, outperforming consultancy methods.
Quick Take
However, when critics and judges have similar classification abilities, debate yields null effects, suggesting a need for careful evaluation of critic capabilities before implementation.
Key Points
- Debate improves judge performance when critics outperform judges in classification tasks.
- Statistically significant gains observed in three out of five model pairings.
- Null effects occur when critic and judge abilities are similar, reducing verification rates.
- Removing rebuttal rounds does not impact judge performance significantly.
- Suggests a cost-effective oversight method for verifiable domains.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 27483v1 Announce Type: new Abstract: Despite theoretical promise, debate as a scalable oversight protocol has produced mixed empirical results: gains in some settings, and null effects in others, especially when the judge does not have information hidden from it. We study proposer-critic debate in a stronger-debater/weaker-judge setting on programmatically verifiable code and logic tasks.
Debate helps the judge over a consultancy baseline when the critic provides a usable advantage: the critic's classification ability must exceed the judge's, and the judge must treat critic speeches as claims to verify rather than testimony to summarize. On the three of five pairings where the condition holds, proposer-critic debate's gains are statistically significant over consultancy, and these pairings are the most capable model pairings.
On the two non-responder pairings in our set, debate produces null effects, and judge verification rates drop by tens of percentage points once a critic enters the transcript. In these cases the critic's binary-classification ability and the judge's are within noise of each other, and the critic's disagreement is parsed as testimony rather than a claim to check.
Ablating rebuttal rounds from debate produces no measurable change in judge performance: a single independent critique recovers the bulk of debate's benefit at lower inference cost. These findings suggest a cheaper primitive for training-free scalable oversight in verifiable domains (answer, critique, judge) and a pre-deployment audit (does the critic beat the judge, and will the judge verify it? ) that predicts when debate will help.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.