RoPoLL: Robust Panel of LLM Judges
Quick Answer
This paper shows that RoPoLL, a robust panel of LLM judges, outperforms traditional LLM jury methods by mitigating bias from individual judges, achieving a 19% improvement on cross-dimensional attacks and significantly outperforming Mistral-Large-3 in specific corruption scenarios.
Quick Take
RoPoLL, a robust panel of LLM judges, outperforms traditional LLM jury methods by mitigating bias from individual judges, achieving a 19% improvement on cross-dimensional attacks and significantly outperforming Mistral-Large-3 in specific corruption scenarios. It utilizes a geometric median for aggregation, ensuring optimal performance against up to 50% corruption rates.
Key Points
- RoPoLL uses a geometric median to aggregate LLM judge scores, enhancing robustness.
- It shows a 19% performance gain on cross-dimensional attacks compared to PoLL.
- A 3-judge RoPoLL committee outperformed Mistral-Large-3 by 1.31x under 30% corruption.
- RoPoLL maintains optimal performance even with up to 50% biased contamination.
- The model addresses issues like mode collapse and safety refusal in LLM evaluations.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 30931v1 Announce Type: new Abstract: The LLM Jury, a Panel of LLM Evaluators (PoLL) reporting consensus scores, has become a practical alternative to single-judge LLM evaluation, yet its statistical behavior remains poorly understood. We formalize the LLM Jury under the Huber contamination model and show that PoLL incurs unbounded bias under any positive contamination, regardless of jury size, whenever a single judge fails in a biased, LLM-typical way (mode collapse, sycophancy, safety refusal).
Framing jury consensus as classical robust mean estimation, we propose RoPoLL (Robust Panel of LLM-as-Judge), which preserves the PoLL panel but replaces the aggregation function with a robust mean estimator, instantiated with the geometric median (GM): tuning-free, with the optimal finite-sample breakdown point 1/2.
A finite-sample error bound and a matching information-theoretic minimax lower bound agree on the parametric rate sigma*sqrt(d/N) and differ on the breakdown floor by a factor of sqrt(d), a statistical-computational gap that polynomial-time RoPoLL pays relative to the intractable Tukey halfspace median.
Across 13 open-weight judges (4B-675B), three reward-model benchmarks, and four corruption regimes at rates up to 50%, RoPoLL dominates PoLL on every biased corruption type: by about 19% on cross-dimensional attacks at matched compute, and by orders of magnitude on heavy-tailed Byzantine adversaries. A 3-judge RoPoLL committee at 38B beats Mistral-Large-3 (675B) by 1.
31x on HelpSteer-2 under 30% bimodal-random corruption, an 18x parameter advantage at better accuracy; a Noisy-GT control confirms the premium is paid against biased contamination, not benign imprecision.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Verification Horizon: No Silver Bullet for Coding Agent Rewards
As coding agents evolve, verifying solutions becomes more challenging than generating them, necessitating a focus on scalable, faithful, and robust verification methods. The study reveals that no fixed reward function can sustain effectiveness as model capabilities advance, emphasizing the need for verification to evolve alongside solution generation.